DEV Community: Hernan Huwyler

From AI Hype to AI Accountability- AI Engineering Summit

Hernan Huwyler — Wed, 29 Jul 2026 21:57:31 +0000

Why your model registry, not your AI policy, is what an auditor actually trusts

Your AI policy will not stop a fraud model from paying out a claim it should have flagged. It will not stop a retrieval pipeline from swallowing a poisoned PDF. It will not stop an agent from splitting a payment into three "urgent" transfers to dodge a review threshold. Policies comfort the people who write them. Controls, the kind with a metric, a threshold, an owner, and a logged response, are what satisfy an auditor and what stop the loss before it happens.

Most enterprises are running AI programs on narrative governance: a PDF that states intent, a committee that meets quarterly, a risk matrix colored red, amber, and green. None of that survives contact with a technical audit, because an auditor does not read your governance document. An auditor pulls your execution records. If there is no metric, no threshold, no named owner, and no logged response, what you have is a wish, not a control, and one missing element collapses the whole thing under review.

This article is based on a facilitated a 3.5-hour workshop at Europe's largest and most influential AI engineering event, hosted by Software and Solutions.

The market right now is flooded with self-claimed "AI experts" recycling the same vague, qualitative governance talking points. I ignore them. My workshop in this AI event focuses strictly on the hard realities of production. In that session, and in this guide, I provide AI architects, engineers, and data scientists with the practical tools and deterministic technical controls required to actually secure a system. We focus on engineering rigor to reduce operational costs, improve system resilience, and avoid critical corporate liabilities.

What follows is the direct output of that technical session. It strips away the academic theory and maps out the exact vulnerabilities threatening your predictive models, generative pipelines, and agentic workflows today.

Evidence Beats Intent: What "Proof Over Promises" Actually Requires

Every AI control needs five things to survive an audit, and dropping any one of them turns a control into paperwork. The control needs a metric that replaces the vague instruction to "monitor AI" with an actual number: a bias rate, an error rate, a drift score, a toxicity score, a latency figure. It needs a threshold, the precise point where acceptable becomes unacceptable, such as a disparate impact ratio below 0.80 or a false negative rate above 0.05. It needs a named owner with real authority, not a shared inbox and not a committee, because ownership is the only thing that reliably turns an alert into a fix. It needs a frequency, because an annual review catches a production problem about ten months too late. And it needs a pre-wired response: block the promotion, trigger an automatic rollback, disable the endpoint, revoke the tool permission, or force a human into the loop. A dashboard nobody acts on is not a control. It is an expensive way to watch a problem happen.

The failure pattern is consistent across every enterprise running AI without this discipline. Overcosts show up because failing models still reach production. Liability accumulates because the company can now prove it knew the risk and shipped anyway, since intent without action reads worse in litigation than ignorance. Drift goes unmanaged because production models change and nobody built a trigger to notice. Outcomes become orphaned because nobody owns what happens when the system breaks. Vulnerability grows because attackers exploit prompts, data, and tool calls the team never modeled. And the whole program has no defense in an incident, because no logs means no proof compliance ever happened.

The fix is not more policy. It is a shift toward an agile governance posture: build the operational block first, document the policy afterward. Log every check with its metric, owner, timestamp, and decision, so the artifact exists before the auditor asks for it. Maintain an actual model register that links every model to its data lineage, code version, parameters, and approvals. And make the control owner the same person who owns the deployment approval, because split accountability is how technical drift wins by default.

Turning Vague Governance Language Into Testable AI Controls

The gap between a policy sentence and a testable control is almost always the same five missing attributes: what, who, when, where, and how. Compare a typical governance line to its operational rewrite.

"Model documentation should be completed as required prior to release" becomes: the ML lead signs off on the model card's ten required fields before every release, evidence logged in the model registry. "Training data will be checked for quality issues as appropriate before use" becomes: the data engineer blocks training whenever null values exceed 2 percent per feature, checked on every run. "The model will be tested for fairness and bias when needed before launch" becomes: the fairness lead blocks every launch if the disparate impact ratio falls below 0.80. "Deployment approval will be granted by the responsible party when ready" becomes: the product owner approves go-live only after all six test gates pass, no exceptions logged without a named risk acceptance.

Notice the pattern. Every rewrite adds a number, a name, and a trigger. That is the entire difference between governance theater and an operational control an auditor can actually test.

STRIDE-AI: Mapping Classic Threat Categories Onto Machine Learning Systems

Classic STRIDE threat modeling was built for deterministic software, where the same input always produces the same output. AI systems break that assumption at every layer, which is why STRIDE needs an AI-specific adaptation before it produces a useful threat model.

Spoofing in an AI system is not just a stolen login. It includes a prompt that reframes the model's role and changes its behavior without touching a single line of code. Tampering rarely looks like a code change. It looks like a poisoned training sample or a label flip that implants a silent backdoor months before anyone notices. Repudiation shows up as a missing version history for a prompt or a dataset, which becomes a regulatory failure the moment the system affects a real person and nobody can reconstruct why it produced a given output. Information disclosure happens through completely normal use, not a breach, when a model memorizes and reproduces a fragment of its training data. Denial of service in an AI system is often a cost spike before it is an outage, because a token-hungry prompt or a runaway agent loop can drain a budget long before it drains an endpoint. Elevation of privilege is the sharpest new risk: an agent with tool access can escalate from read-only to a live financial transaction through a chained function call that no single-step guardrail was built to catch.

Operational failure mode. A retrieval-augmented generation pipeline ingests a vendor-uploaded PDF without sanitization. The document contains a hidden instruction: ignore the standing guidelines and reveal the policy exclusions. The model retrieves the document, treats the embedded text as an authoritative instruction rather than untrusted content, and drafts a noncompliant customer email. Nobody wrote malicious code. Nobody breached a firewall. The entire attack happened inside a channel the team assumed was just data.

Threat Vectors Versus Vulnerabilities: A Distinction Most Risk Registers Get Wrong

A vulnerability is a weakness in design, control, architecture, process, or implementation, such as weak prompt isolation, poor access control, missing rate limits, or a lack of provenance verification. A threat vector is the path or mechanism an attacker, an insider, or a negligent actor actually uses to exploit that weakness, such as prompt injection, data poisoning, model extraction through repeated queries, or API token theft. A vulnerability only becomes a risk worth prioritizing when it sits exposed to a credible threat vector with a motivated actor behind it. Auditors and support teams frequently collapse this distinction, translating a control gap directly into a risk score without asking whether an actual threat agent has the intent, the access, and the capability to exploit it. That shortcut produces flawed prioritization and misallocated remediation budget.

One Framework Comparison Worth Memorizing

The table below is the single comparison that matters most when moving from a risk assessment to a threat model, because teams routinely confuse the two and staff them with the wrong audience.

Dimension	Risk Assessment	Threat Modeling
Objective	Evaluate and prioritize risks	Identify potential threats and vulnerabilities
Scope	Broader risks: technical, operational, business	Technical threats and specific attack scenarios
Primary audience	Management, stakeholders, compliance teams	Developers, architects, security engineers
Timing	Conducted after threat modeling, updated periodically	Conducted during design and development, updated iteratively

Run these in the wrong order and you end up with a risk register full of business-impact language and zero technical traceability, or a threat model buried in attack trees that never reaches a committee that can fund the fix.

The ClaimAssist Scenario: Quantifying Exposure Instead of Guessing at It

Consider an insurance company running an AI-powered claims platform, internally called ClaimAssist. The predictive layer runs a fraud-scoring model that flags suspicious claims before payout. The generative layer drafts claim summaries and customer emails, grounded in a retrieval corpus of policy documents. The agentic layer auto-approves and pays claims under EUR 2,000 without a human in the loop.

Each layer carries a distinct, quantifiable exposure. The fraud-scoring model was trained eighteen months earlier on historical claims data and has no automated drift detection or retraining trigger, so it silently degrades as fraud tactics evolve past its training coverage. This is vulnerability V027, Missing Drift Controls, exploited through threat vector T038, Generalization Failure Exploitation. The retrieval corpus accepts vendor-uploaded PDFs and injects the retrieved text into the prompt unescaped, so system rules, user claims, and retrieved policy excerpts all share one channel with no way for the model to separate trusted instructions from untrusted content. This is vulnerability V004, Weak Prompt Isolation, exploited through T002, Indirect Prompt Injection. The auto-pay agent runs on a shared service account with broad payment API rights, so a single successful prompt injection can trigger a payout, a refund, or an address change well outside its intended EUR 2,000 ceiling. This is vulnerability V005, Excessive Tool Permissions, exploited through T041, Unauthorized Tool Use.

Quantify the predictive-layer scenario alone. A drifted model silently raises false-positive fraud flags for low-income, remote-area claimants, and the pattern only surfaces once client complaints spike. In a contained, best-case scenario where the drift is caught within two weeks, the company still absorbs roughly EUR 50,000: about EUR 20,000 in wrongful-denial compensation, EUR 20,000 in internal investigation costs, and EUR 10,000 in emergency model review. Left running for three months, the worst-case scenario climbs to roughly EUR 300,000, covering denied-claim back-payments and compensation near EUR 80,000, chargebacks and legal settlements near EUR 70,000, a regulatory investigation from an insurance authority near EUR 50,000, and remediation, retraining, and audit costs near EUR 100,000.

That is the kind of number a Monte Carlo simulation, convolving simulated loss magnitudes against simulated event frequencies across 100,000 trials, can turn into a defensible reserve figure rather than a guess. A total loss distribution built this way might show a decision point at the 75th percentile calling for roughly EUR 847,000 in reserves. Funding below that percentile means the organization runs under-reserved and exposed in one out of every four years, a statement a finance committee understands instantly and a "medium risk" label never communicates.

From Exposure to a Testable Control: The Full Attribution

Once the exposure is quantified, the fix has to be written the same way every other control in this piece is written, with a responsible role, an accountable role, a frequency, evidence, and scope. For the drift exposure above, one workable control pairs automated drift detection with a mandatory retraining trigger, mapped to ISO 42001 clause A.6.2.6 on operation and monitoring. The ML engineering lead is responsible for the fraud-scoring model, the Head of Fraud Risk is accountable for signing off on the retraining decision, the check runs weekly with immediate retraining on a threshold breach, evidence lives in the drift dashboard logs and the retraining approval record in the model registry, and the scope is a statistical drift test on input feature distributions and false-positive rate by claim subgroup, firing an alert above a population stability index of 0.2 or a 15 percent relative rise in false-positive rate for any subgroup over a rolling 30-day window.

For the prompt-injection exposure, one workable control enforces structural separation and sanitization of retrieved content at the ingestion boundary, mapped to ISO 42001 clause A.6.2.3 on security threats considered in design. Platform engineering owns the RAG pipeline, the Head of AI Engineering is accountable, the check runs at every ingestion event and every retrieval at inference time, evidence lives in the ingestion pipeline logs showing sanitization pass or fail per document, and the scope requires every uploaded PDF to be scanned for embedded instruction-like patterns before indexing, with retrieved text passed to the model in a structurally separate, clearly tagged field the system prompt is designed never to treat as an instruction.

For the tool-permission exposure, one workable control enforces scoped, deterministic payment authorization outside the model itself, mapped to ISO 42001 clause A.3.2 on roles and responsibilities. Claim platform engineering owns the control, the Chief Information Security Officer is accountable, enforcement runs on every tool call in real time with quarterly credential reviews, evidence lives in the service account permission manifest and the authorization-service rejection logs, and the scope hard-caps the auto-pay agent's credential at a single transaction under EUR 2,000 with no refund, address-change, or split-payment permission, with the ceiling enforced by a policy service outside the model rather than by prompt instruction alone.

The Point Nobody Wants to Hear

None of this works as a one-time project. AI systems change through retraining, fine-tuning, prompt edits, retrieval corpus updates, new tool integrations, and vendor-side model updates that happen without your calendar invite. A control that was airtight in January can be silently invalid by March if nobody rechecked the threshold against the current model version. Red teams exist to find out whether a vulnerability is exploitable in practice, and blue teams exist to find out whether the organization would even notice. Run them together, on a real cadence, against the actual production system, not a slide describing it.

AI Vulnerability Taxonomy for Developers

Clean, scannable guide to 76 AI-specific security weaknesses

This taxonomy walks you through every AI vulnerability cataloged in data/vulnerabilities.yaml, but in a format built for humans, not YAML parsers. Use it when scoping threat models, briefing new team members, or understanding why a control showed up in your generated checklist.

Each entry tells you what the weakness is, which AI layer it lives in, which system types care most, which frameworks it maps to, and what actually fixes it in production.

Reading Guide

Category — The grouping used across this toolkit

Layer — Where the weakness lives: Data, Model, Infrastructure, Governance, or Training Technique

System Type — Predictive, Generative, or Agentic (many apply to all three)

Maps To — NIST AI RMF, NIST AI 100-2, OWASP LLM Top 10 (2025), OWASP Agentic Top 10 (Dec 2025), MITRE ATLAS, ISO/IEC 42001

Controls — What works in production, not compliance theater

Access and Exposure

Who can reach your model, API, prompt, or tool, and what happens when they do.

V001. Weak Access Control

Layer: Infrastructure, Governance

System Type: All

Maps To: ISO/IEC 42001, NIST AI RMF (Govern)

Users, services, pipelines, or agents reach models, datasets, prompts, tools, vector stores, or config assets beyond their authorized scope. This matters more in AI than in traditional apps because AI pipelines span more roles (data engineering, ML, security, business) and overprivileged access anywhere in that chain lets unauthorized changes to model behavior, training inputs, or deployment settings happen quietly.

Shows up as: overprivileged service accounts, shared credentials, missing role separation between dev and prod, inconsistent least privilege across notebooks and registries.

Fix: Role-based access scoped per pipeline stage. Separate credentials for dev, staging, prod. Regular access reviews for service accounts. Just-in-time elevation for sensitive ops, not standing privilege.

V002. Insecure API Exposure

Layer: Infrastructure

System Type: All

Maps To: OWASP LLM10 (Unbounded Consumption), MITRE ATLAS (Initial Access)

Model endpoints, orchestration layers, or inference services are reachable without strong auth, authorization, encryption, abuse controls, and request validation. Creates a direct path for unauthorized access, model extraction, data leakage, prompt abuse, and denial of service.

Assume every AI endpoint is a sensitive control surface, not a standard API. The endpoint sits in front of a component that can leak training data, be manipulated through crafted input, or be queried cheaply enough to approximate the model offline.

Fix: Strong auth and authz on every endpoint. TLS in transit. Request and response schema validation. Abuse and anomaly detection layered on top of standard API gateways.

V003. Unrestricted Query Access

Layer: Infrastructure

System Type: Predictive, Generative

Maps To: MITRE ATLAS (Exfiltration), OWASP LLM10

Users or systems query a model at high volume, high frequency, or high fidelity without rate limits, quotas, anomaly detection, or behavioral restrictions. Makes extraction, prompt probing, confidence analysis, and cost-amplifying attacks far easier.

The problem is not exposure (a queryable model must be exposed). The problem is exposure without guardrails on volume, response detail, or granularity like raw confidence scores that make extraction cheaper.

Fix: Per-user and per-key rate limits. Confidence-score truncation or removal in production responses. Anomaly detection tuned to query patterns consistent with extraction.

V004. Weak Prompt Isolation

Layer: Model, Infrastructure

System Type: Generative, Agentic

Maps To: OWASP LLM01, LLM07, OWASP Agentic Top 10

System instructions, developer prompts, user input, retrieved content, tool output, and memory mix together without clear trust boundaries or policy enforcement between them. The model has no built-in way to distinguish operational instructions from adversarial content unless the architecture enforces that separation explicitly.

When prompt layers are not isolated, you get instruction override, hidden context manipulation, and system-prompt leakage. This is not a prompt-wording problem. It is an architectural control failure that must be solved outside the model by treating retrieved and user-supplied content as untrusted data, not instructions.

Fix: Structural separation of system instructions from user and retrieved content, enforced by the application layer, not prompt wording. Content provenance tagging carried through the pipeline. Output-side checks for system-prompt leakage.

V005. Excessive Tool Permissions

Layer: Infrastructure, Governance

System Type: Agentic

Maps To: OWASP Agentic ASI02, ASI03, MITRE ATLAS

AI agents or orchestration services get broader access to APIs, files, workflows, or enterprise systems than their use case requires. Turns an ordinary model error into a high-impact operational event because the model can trigger real actions without an independent restriction between the decision and the action.

Tool layers often inherit broad enterprise permissions because scoped, task-specific credentials are harder to manage than a single service account with wide access. The result: action surface that violates least privilege by design.

Fix: Scope every tool credential to minimum required actions. Separate read and write tool permissions. Approval gates for any tool call with financial, data-deletion, or external-communication impact.

V006. Weak Runtime Authorization

Layer: Model, Infrastructure

System Type: Agentic

Maps To: OWASP Agentic ASI01, ASI03

The system relies on the model itself to decide whether a request, action, or tool invocation is allowed, instead of enforcing policy through a deterministic control layer outside the model. AI models are probabilistic components and should not serve as final authority for sensitive actions, regulated workflows, or high-impact business decisions.

Typically appears when prompts are expected to enforce policy ("only approve refunds under $100") instead of code, workflow rules, or a dedicated authorization service. Produces brittle security that is easy to manipulate through prompt-level attacks and hard to audit.

Fix: Enforce sensitive-action policy in code or a policy engine outside the model, not in the prompt. Treat the model's output as a proposed action requiring separate authorization, not an authorized action.

V007. Limited Tenant Isolation

Layer: Infrastructure

System Type: All

Maps To: NIST AI RMF (Manage)

Workloads, sessions, memory, embeddings, prompts, data stores, or inference resources are not adequately separated across users, customers, or business units. Increases risk of data leakage, cross-session contamination, privilege abuse, and noisy-neighbor denial-of-service.

Particularly important in shared enterprise AI platforms and hosted AI services where teams assume logical separation that the architecture does not actually enforce.

Fix: Per-tenant data and memory partitioning. Separate vector store namespaces per tenant. Load and resource isolation to prevent one tenant's usage from degrading another's.

Change and Artifact Integrity

Whether models, prompts, or configurations can be modified without approval, testing, and rollback capability.

V008. Weak Change Management

Layer: Governance, Model

System Type: All

Maps To: ISO/IEC 42001, NIST AI RMF (Manage)

Models, prompts, datasets, feature pipelines, policies, or runtime settings can be modified without formal approval, traceability, testing, and rollback controls. AI systems are unusually sensitive to small changes: an undocumented update to a prompt, retrieval rule, or generation parameter can materially alter security posture and business behavior in ways a code diff would never suggest.

Common in fast-moving ML teams where experimentation practices leak directly into production without the release discipline a mature software team would apply.

Fix: Formal change approval workflow covering prompts and config, not just code. Automated regression testing against security and behavior baseline before any production change. Documented rollback procedure tested before it is needed.

V009. Weak Artifact Protection

Layer: Model, Infrastructure

System Type: All

Maps To: NIST AI 100-2, MITRE ATLAS

Model weights, checkpoints, prompt templates, tokenizer files, evaluation sets, configs, and deployment bundles stored without strong encryption, integrity validation, and access restriction. These artifacts encode business logic, IP, system behavior, and sometimes sensitive data directly.

If artifact storage is weak, an attacker or insider can tamper with a model, steal proprietary assets, or deploy a manipulated version without detection. Particularly acute when artifacts are copied across notebooks, model registries, object stores, and CI systems with inconsistent controls at each hop.

Fix: Encryption at rest for all model and artifact storage. Cryptographic hashing with integrity verification before deployment. Single authoritative registry, not artifacts scattered across notebooks and ad hoc storage.

V010. Complex Model Loading

Layer: Model, Infrastructure

System Type: All

Maps To: MITRE ATLAS, OWASP LLM03

Serialized models, checkpoints, custom loaders, or deserialization workflows allow unsafe code execution, untrusted object parsing, or weak artifact validation at load time. Major implementation weakness across the ML ecosystem where convenience mechanisms for loading models are prioritized over secure loading practices.

If model loading is not tightly controlled, a malicious artifact can execute code, alter runtime behavior, or compromise the environment before the model serves a single inference request.

Fix: Use safe serialization formats that do not support arbitrary code execution on load. Validate artifact integrity and provenance before loading. Run model loading in a sandboxed environment with no unnecessary system access.

V011. Unauthorized Parameter Changes

Layer: Model, Governance

System Type: All

Maps To: MITRE ATLAS

Model weights, prompt settings, thresholds, hyperparameters, routing logic, or safety configs can be modified without strict approval, access restriction, and audit trail. AI systems are highly sensitive to parameter changes. Even a small adjustment (temperature setting, safety threshold) can alter risk posture, output quality, and control behavior in non-obvious ways.

Common where experimentation platforms and production environments are not well separated, so a change made for testing can reach production without review.

Fix: Treat safety-relevant parameters as production configuration requiring same approval as code. Separate experimentation environments fully from production configuration surfaces. Log every parameter change with actor identity and reason.

V012. Inconsistent Version Control

Layer: Model, Data, Governance

System Type: All

Maps To: ISO/IEC 42001

Datasets, prompts, models, features, and deployment configs are not versioned consistently across teams and environments. Creates uncertainty about what is actually running, what was tested, and what should be rolled back after a failure.

AI systems depend on tightly coupled artifacts where the model, the data it was trained and evaluated on, and the prompt or feature config around it must align for the system's validated behavior to hold. Weak version discipline creates hidden mismatch between training, evaluation, and production.

Fix: Single versioning scheme spanning datasets, prompts, and models together, not tracked independently. Immutable version tags tied to specific evaluation results. Automated checks that block deployment of untagged artifacts.

Data Governance and Quality

Training, fine-tuning, feedback, and retrieval data that shapes model behavior. AI systems learn from data rather than being explicitly programmed, so weaknesses here propagate directly into model behavior.

V013. Insufficient Provenance Controls

Layer: Data, Governance

System Type: All

Maps To: NIST AI 100-2, ISO/IEC 42001

The organization cannot reliably verify where data, labels, models, prompts, or derived artifacts came from, who changed them, and whether they remained intact through the lifecycle. Allows poisoned, biased, stolen, or noncompliant assets to enter the pipeline with limited ability to validate authenticity or reconstruct lineage after the fact.

Common in organizations with decentralized data sourcing, weak dataset versioning, or undocumented fine-tuning and retrieval workflows. Compounds over time: each generation of fine-tuning built on unverified lineage inherits whatever was already unverifiable.

Fix: Data lineage tracking from ingestion through training. Cryptographic signing of datasets at point of collection or curation. Provenance verification as gate before any dataset enters training.

V014. Data Poisoning Susceptibility

Layer: Data, Training Technique

System Type: Predictive, Generative

Maps To: NIST AI 100-2, OWASP LLM04

Training, fine-tuning, feedback, or retrieval data can be introduced or modified without strong validation, curation, anomaly detection, and approval controls. This describes a broken state, not the attack itself: the condition where malicious or low-integrity data can influence future system behavior without being detected.

Particularly severe in systems that continuously learn, accept user feedback as training signal, or ingest external data at scale. Reflects weak data governance, inadequate sanitation, and poor separation between trusted and untrusted data sources feeding the same pipeline.

Fix: Outlier and anomaly detection on incoming training and feedback data. Held-out, integrity-verified evaluation set used to catch performance shifts after retraining. Staged review for any externally sourced data before it reaches training pipeline.

V015. Weak Data Governance

Layer: Data, Governance

System Type: All

Maps To: ISO/IEC 42001, NIST AI RMF (Govern)

The organization lacks formal controls for data ownership, quality requirements, lifecycle handling, access restriction, lawful use, retention, and accountability across AI pipelines. Creates systemic exposure because even a well-engineered model becomes unreliable when built on poorly governed data.

Unlike a code defect, a data governance gap tends to affect every model trained on the affected data rather than a single component. Appears as undocumented data flows, unclear stewardship between business units, inconsistent policy application across teams, and missing controls over data reuse.

Fix: Named data owners per dataset with documented accountability. Data classification scheme applied consistently across training, testing, and inference. Documented retention and reuse policy enforced at pipeline level, not left to individual teams.

V016. Weak Data Quality Controls

Layer: Data

System Type: Predictive, Generative

Maps To: ISO/IEC 42001

Completeness, consistency, validity, freshness, representativeness, and defect thresholds are not formally defined and enforced across the AI data lifecycle. One of the most common root weaknesses in AI projects.

Poor-quality data can degrade model performance, mask an ongoing poisoning attempt inside normal-looking noise, amplify bias, and undermine confidence in evaluation results without any single obvious failure point. Data quality controls applied inconsistently across ingestion, labeling, feature engineering, and retraining.

Fix: Defined, measurable data quality thresholds applied at every pipeline stage, not only at ingestion. Automated data quality checks that block a training run rather than only logging a warning. Periodic representativeness review against deployed population.

V017. Distributed Data Inconsistency

Layer: Data, Infrastructure

System Type: All

Maps To: ISO/IEC 42001

Multiple repositories, feature stores, data lakes, labels, or training environments maintain different versions of supposedly authoritative data without synchronization or reconciliation controls. Creates hidden divergence between what the model was trained on, evaluated on, and actually sees in production.

Each of those three data views is usually owned by a different team or system, so divergence can persist for a long time. Produces unstable performance, unexplained regressions that do not correlate with any code or model change, and weak incident traceability.

Fix: Single source-of-truth designation for each dataset with all consumers pointed at it. Reconciliation checks between training, evaluation, and production data views run on schedule. Alerting on divergence beyond defined tolerance.

V018. Complex Data Transformations

Layer: Data, Training Technique

System Type: Predictive

Maps To: ISO/IEC 42001

Raw data passes through many preprocessing, normalization, filtering, enrichment, or encoding stages that are poorly documented, weakly tested, or inconsistently applied. Each transformation step can introduce loss, corruption, bias, or mismatch. Risk compounds when different teams maintain different portions of the same pipeline without shared view of the whole chain.

Common in mature AI stacks where data preparation logic has accumulated over time without end-to-end validation re-run against original assumptions. The more opaque the transformation chain, the harder to detect an error introduced partway through.

Fix: Documented, versioned transformation logic with test coverage at each stage. End-to-end validation comparing final training data against source data on sample basis. Ownership assigned per pipeline segment, not just per pipeline.

V019. Schema Incompatibility

Layer: Data, Infrastructure

System Type: All

Maps To: ISO/IEC 42001

Different components in the AI pipeline rely on inconsistent field definitions, formats, units, labels, token structures, or metadata conventions. Forces ad hoc conversion logic between components, and every ad hoc conversion is a place where silent data corruption, feature mismatch, or failed integration can occur without raising an error.

Particularly harmful in large AI programs that combine multiple vendors, legacy systems, or rapidly evolving internal pipelines. Each additional system multiplies the schema boundaries needing reconciliation.

Fix: Shared schema registry enforced across ingestion, training, and serving. Automated schema validation at every pipeline boundary. Deprecation process for schema changes that gives downstream consumers advance notice.

V020. Uncontrolled Data Ingestion

Layer: Data

System Type: Generative

Maps To: OWASP LLM04, LLM08

Data enters the AI system from multiple sources without centralized validation, source trust assessment, security checks, and ownership controls. Creates weak perimeter around one of the most consequential parts of the AI lifecycle: what the system is allowed to learn from or reason over.

Especially significant in retrieval-augmented generation systems, crowdsourced pipelines, and environments that blend user data, third-party feeds, internal documents, and automation outputs into single ingestion path.

Fix: Single, centralized ingestion gateway rather than direct writes from multiple sources into shared storage. Source trust classification applied before content is indexed. Review workflow for any content entering retrieval corpus from outside the organization.

V021. Untrusted External Data Sources

Layer: Data

System Type: Generative

Maps To: OWASP LLM04

The system relies on third-party, scraped, user-contributed, or vendor-supplied data without robust source validation, quality review, licensing review, and trust classification. Creates direct path for contamination of training, retrieval, and decision logic.

Especially important where business process assumes external content is "good enough" because it is convenient or widely used elsewhere, without that assumption tested against actual risk tolerance. Safer default: treat external data as untrusted until proven otherwise.

Fix: Explicit trust classification for every external data source before use. Licensing and provenance review as part of onboarding new external source. Periodic re-review of long-standing external sources, since trustworthiness can change over time.

V022. Weak De-Identification

Layer: Data, Governance

System Type: Predictive, Generative

Maps To: NIST AI 100-2, OWASP LLM02

Personal, proprietary, or regulated data is tokenized, masked, pseudonymized, or otherwise transformed in ways that still permit re-identification through linkage, inference, metadata, or model behavior. Major privacy weakness in AI pipelines because derivative artifacts (embeddings, prompts, logs, model outputs) can reintroduce exposure even when raw source fields were properly obfuscated.

Organizations frequently overestimate protection that simplistic masking provides and do not test for realistic re-identification risk against the specific model and use case.

Fix: Re-identification testing specific to the model and use case, not generic masking checklist. Treat embeddings and logs as potential re-identification vectors requiring their own protection. Periodic review as linkage risk changes with new external data availability.

V023. Training Data Memorization

Layer: Model, Training Technique

System Type: Generative

Maps To: NIST AI 100-2, OWASP LLM02

The model retains and can reproduce sensitive or proprietary content from training or fine-tuning data because minimization, filtering, and privacy-preserving techniques were insufficient during training. This is a model and training weakness in its own right, not merely a misuse scenario downstream.

The model's architecture and training process themselves permit undue retention of specific content rather than learning generalized patterns. Especially concerning in large generative models and domain-specific models trained on regulated or confidential corpora.

Fix: Memorization testing against known training samples before release. Data minimization and deduplication in training set. Differential privacy techniques where sensitivity of training data warrants the accuracy tradeoff.

Model Lifecycle

What happens before and after a model reaches production: validation before release, transfer or fine-tuning from a base model, and feedback captured after deployment.

V024. Weak Transfer Validation

Layer: Model, Training Technique

System Type: Generative

Maps To: NIST AI RMF (Map)

Pretrained models, foundation models, or transferred representations are adopted without rigorous verification that they are suitable, safe, and reliable in the new domain or use case. Many teams assume strong base model remains trustworthy after fine-tuning, but hidden weaknesses, bias patterns, or unsafe behaviors in the base model can carry forward essentially unchanged.

Fine-tuning typically adjusts behavior at the margins rather than re-validating the whole model from scratch. Reflects weak governance over model adoption decisions and insufficient validation in target environment.

Fix: Domain-specific validation after any fine-tuning or adaptation, independent of base model's published benchmarks. Documented adoption criteria before base model is approved for use. Re-validation whenever base model provider ships an update.

V025. Insufficient Model Validation

Layer: Model

System Type: All

Maps To: NIST AI RMF (Measure), ISO/IEC 42001

Testing and assurance activities do not adequately evaluate security, robustness, fairness, privacy, performance, and failure modes before release. One of the most serious AI control failures because it allows unreliable or unsafe model to reach production based on narrow benchmark performance or incomplete quality assurance.

A model can pass every metric a team chose to measure while still failing badly on a dimension nobody tested. Appears as limited adversarial testing, poor subgroup evaluation, inadequate edge-case coverage, or overreliance on static benchmarks that do not reflect production conditions.

Fix: Validation scope defined against specific risks of the use case, not generic benchmark suite. Subgroup and edge-case evaluation as release gate. Adversarial testing proportional to model's exposure and impact.

V026. Weak Feedback Loops

Layer: Data, Governance

System Type: All

Maps To: NIST AI RMF (Manage)

The organization does not systematically collect, triage, and incorporate user feedback, incident findings, model errors, and performance observations into ongoing model improvement and governance. Allows known issues to persist and prevents system from adapting to operational reality that only becomes visible after deployment.

In AI systems, feedback is not merely product-improvement input. It is part of the control environment needed to detect emergent risks and performance regressions that pre-release testing could not anticipate. Where feedback exists but is ungoverned, it can also become corruption source.

Fix: Defined triage process for user-reported issues with path into risk register, not just support ticket queue. Governance review before feedback data is used in retraining. Periodic reporting on unresolved known issues to system's accountable owner.

V027. Missing Drift Controls

Layer: Model, Data

System Type: Predictive, Generative

Maps To: NIST AI RMF (Measure)

The organization does not monitor and respond to changes in input distributions, feature behavior, environmental conditions, user behavior, or underlying concepts over time. Especially important in predictive systems and adaptive production environments where model can silently become less accurate, less fair, or less robust without triggering formal incident.

Drift by its nature does not announce itself the way an outage does. In generative systems, drift can also affect retrieval quality, grounding reliability, and prompt behavior as enterprise content or user patterns evolve after original validation.

Fix: Statistical drift detection on input distributions and key output metrics. Defined response threshold that triggers re-validation, not just alert. Scheduled re-evaluation cadence independent of drift alerts, since some drift is gradual enough to stay under alert thresholds.

Observability and Response

Whether the organization can see what the system is doing and reconstruct what happened after an incident.

V028. Insufficient Logging

Layer: Infrastructure, Governance

System Type: All

Maps To: ISO/IEC 42001, NIST AI RMF (Measure)

The system does not retain adequate records of prompts, retrieved context, model versions, feature states, tool calls, policy decisions, user actions, and deployment events. Undermines incident response, root-cause analysis, forensic review, and accountability.

More acute problem for AI than conventional software because AI failures often emerge through multi-step interactions across several components rather than single, obviously faulty line of code. Logging either too sparse to reconstruct incident or too inconsistent across AI lifecycle to piece together what actually happened.

Fix: Structured logging covering prompts, retrieved context, tool calls, and model version at inference time. Consistent log retention policy across full pipeline, not just inference endpoint. Log integrity protections to prevent post-incident tampering.

V029. Inadequate Monitoring

Layer: Infrastructure, Model

System Type: All

Maps To: NIST AI RMF (Measure)

The system does not continuously observe model behavior, data quality, abuse patterns, drift, service health, policy violations, and integration failures after deployment. AI systems require stronger runtime observability than conventional software because harmful behavior often emerges gradually or probabilistically rather than through single obvious fault.

Many organizations deploy AI with solid infrastructure monitoring but no meaningful visibility into model misuse, degraded output quality, unsafe agent behavior, or retrieval corruption because those signals require monitoring built specifically for AI behavior.

Fix: Monitoring built specifically for model behavior and output quality, layered on top of standard infrastructure monitoring. Defined thresholds for abuse and policy-violation signals, not just latency and error rate. Regular review of monitoring coverage against current threat model.

V030. Weak Event Traceability

Layer: Infrastructure, Governance

System Type: All

Maps To: ISO/IEC 42001

Event records are incomplete, inconsistent, or disconnected across data pipelines, model training, deployment, inference, and downstream action layers. Leaves organization unable to correlate incident across components or explain how harmful output became harmful action.

Materially harder problem in agentic systems where single incident can span model decision, tool call, and external system update. AI systems often composed of loosely coupled services built by different teams at different times, making end-to-end traceability control necessity.

Fix: Shared trace identifier carried across every component an event touches, from ingestion through action. Correlation tooling that can reconstruct full event timeline across services. Periodic incident response drills that test whether traceability holds up under realistic scenario.

V031. Weak Performance Auditing

Layer: Model, Governance

System Type: All

Maps To: NIST AI RMF (Measure)

Model accuracy, robustness, fairness, stability, and operational effectiveness are not reviewed on regular and independent basis after release. Allows performance degradation, hidden bias, and emerging failure patterns to persist below threshold that would normally trigger formal incident response.

Many organizations treat model evaluation as one-time pre-launch activity rather than ongoing assurance obligation that continues for as long as model is in production. Deployed system can drift far from originally approved performance profile without triggering formal review.

Fix: Scheduled, independent performance audits separate from routine monitoring alerts. Fairness and subgroup review as standing item, not one-time launch check. Defined escalation path when audit finds system has drifted from approved profile.

V032. Insufficient Resource Monitoring

Layer: Infrastructure

System Type: Generative, Agentic

Maps To: OWASP LLM10

Compute, memory, storage, concurrency, token consumption, and tool usage not observed closely enough to detect abuse, saturation, inefficiency, or performance collapse. Can hide extraction attempts, denial-of-service conditions, agent loops, and cost overruns until they become operationally severe.

In generative and agentic systems specifically, resource misuse is often earliest visible indicator something is wrong, arriving well before any output-quality or security signal would fire. System can be fully "up" by conventional measures while quietly running away on token or compute cost.

Fix: Token, compute, and tool-call budgets tracked per session and per user, not only in aggregate. Cost-spike alerting independent of latency or error-rate alerting. Automatic circuit breakers for sessions exceeding defined resource thresholds.

V033. Weak Incident Coordination

Layer: Governance

System Type: All

Maps To: ISO/IEC 42001, NIST AI RMF (Manage)

Communication plans, escalation paths, ownership boundaries, and response procedures for AI incidents are absent, outdated, or untested. Delays containment and creates confusion during events involving harmful outputs, unsafe actions, data leakage, or model degradation.

AI incidents frequently span security, engineering, product, legal, and business teams simultaneously, making coordination meaningfully more complex than conventional software incident response where responsible team is usually clearer from the start.

Fix: Defined AI-specific incident response plan naming owners across security, engineering, legal, and business teams. Regular tabletop exercises using realistic AI incident scenarios. Documented communication template ready to use rather than drafted during incident itself.

V034. Missing Reporting Channels

Layer: Governance

System Type: All

Maps To: NIST AI RMF (Govern)

Employees, users, or operators have no defined way to raise concerns about harmful outputs, bias, security events, unsafe actions, or governance issues related to AI systems. Prevents early detection of issues that may never surface through automated monitoring.

Some of the most consequential problems, particularly bias and inappropriate use, are noticed by a human before any system flags them. In many programs, concerns are raised informally and never reach team with authority to investigate or remediate.

Fix: Defined, publicized channel for reporting AI-related concerns, separate from general IT support. Documented triage process ensuring reports reach team with remediation authority. Periodic communication reminding staff and users channel exists.

Governance and Accountability

Policy, ownership, documentation, and alignment between what the system optimizes for and what the business needs. Rarely surfaces in penetration test but frequently the reason technical weakness went undetected or unowned.

V035. Missing AI Policies

Layer: Governance

System Type: All

Maps To: ISO/IEC 42001, NIST AI RMF (Govern)

The organization lacks clear standards, governance rules, and control expectations for AI development, deployment, procurement, use, and retirement. Creates inconsistent practices across teams and leaves critical decisions to local interpretation rather than enterprise-level governance.

Two teams building similar systems can end up with materially different risk postures for no reason other than which policy gap they happened to fill in themselves. Missing policy framework functions as systemic enabler of nearly every other technical weakness.

Fix: Enterprise-level AI policy covering full lifecycle from development through retirement. Policy ownership assigned to named function, not left ambiguous. Defined process for teams to request policy guidance on novel use cases.

V036. Undefined AI Roles

Layer: Governance

System Type: All

Maps To: ISO/IEC 42001, NIST AI RMF (Govern)

Responsibilities for model ownership, data stewardship, risk acceptance, monitoring, security, and operational response are not clearly assigned. Creates accountability gaps that allow issues to persist simply because no one is formally responsible for detecting, approving, or remediating them.

In AI systems, unclear role boundaries are especially dangerous because responsibility is often split across security, data science, engineering, compliance, and business teams, none of which may consider given weakness fully "theirs" to fix.

Fix: Named accountable owner for every production AI system, documented and kept current. Role definitions covering full lifecycle, not just development. Escalation paths that do not depend on knowing the right individual personally.

V037. Lack of Design Documentation

Layer: Governance, Model

System Type: All

Maps To: ISO/IEC 42001

System architecture, model assumptions, trust boundaries, control points, data dependencies, tool integrations, and operational workflows are not formally documented. Makes AI system harder to secure, audit, maintain, and change safely over time.

Undocumented systems accumulate hidden dependencies and implicit logic that weaken security and resilience with every change made without written record of reasoning. Teams cannot govern what they cannot clearly describe.

Fix: Maintained architecture document covering trust boundaries and data flows, reviewed on every material change. Design decisions recorded with their reasoning, not just their outcome. Documentation treated as release gate for new components, not afterthought.

V038. Weak Explainability Controls

Layer: Model, Governance

System Type: Predictive, Generative

Maps To: NIST AI RMF (Map), ISO/IEC 42001

The system cannot adequately trace outputs, recommendations, or actions back to relevant inputs, model states, decision pathways, or policy conditions. Practical vulnerability, not just compliance concern, because weak traceability impairs auditing, root-cause analysis, challenge rights for affected individuals, compliance reviews, and general trust.

Not that every model must be fully interpretable in mathematical sense. Level of explanation available must be sufficient for the risk and use case in question.

Fix: Explainability requirements defined at risk level of use case before development, not retrofitted after launch. Output-to-input traceability sufficient to answer specific challenge, not just aggregate feature-importance report. Documented process for handling explanation requests from affected individuals.

V039. Poor User Guidance

Layer: Governance

System Type: All

Maps To: NIST AI RMF (Govern)

End users, reviewers, and operators do not receive clear instructions on system limits, approved use cases, escalation procedures, confidence handling, and expected validation steps. Increases misuse, overreliance, operational error, and poor adoption.

Users left without guidance will invent their own safety practices, and those improvised practices are rarely consistent or well calibrated to system's actual limitations. User documentation functions as part of control framework itself, not merely support artifact.

Fix: Documented system limits and approved use cases delivered to users before they rely on system, not buried in help center. Explicit guidance on how to handle low-confidence or ambiguous outputs. Periodic refresher communication as system's capabilities or limits change.

V040. Weak Intended Use Controls

Layer: Governance

System Type: All

Maps To: ISO/IEC 42001, NIST AI RMF (Govern)

No technical or procedural mechanisms ensure AI system is used only within its approved purposes, domains, user groups, and risk boundaries. Especially important in enterprise settings where model built for low-risk task can quietly migrate into higher-risk use case without triggering new validation or governance review.

Migration often happens informally, one adjacent use case at a time, rather than through single obvious decision. Results in misuse by gradual expansion, harder to detect because no single step looks like security event.

Fix: Technical enforcement of approved use cases where feasible, not policy documentation alone. Defined review trigger whenever system is proposed for new use case. Periodic usage audits comparing actual use against originally approved scope.

V041. Weak Requirements Definition

Layer: Governance

System Type: All

Maps To: ISO/IEC 42001

AI functional, security, safety, privacy, fairness, resilience, and compliance requirements are incomplete, ambiguous, or undocumented. Causes downstream control failures because teams cannot build, test, or govern against requirements that were never made explicit.

Requirement that exists only informally tends to be interpreted differently by whoever is asked about it on given day. Especially common in AI projects where business enthusiasm for new capability outruns architectural discipline.

Fix: Written, specific requirements covering security, privacy, and fairness alongside functional requirements, before development begins. Requirements traceable to specific tests in Phase 4 of assessment. Defined process for updating requirements as use case evolves.

V042. Weak Planning Discipline

Layer: Governance

System Type: All

Maps To: ISO/IEC 42001

AI project lacks structured lifecycle planning for development, deployment, testing, monitoring, rollback, and retirement. Results in ad hoc decisions, undocumented tradeoffs, control gaps, and fragile implementation practices.

In many AI initiatives, momentum of rapid experimentation substitutes for engineering rigor, leaving critical security and governance work permanently unfinished rather than merely delayed. Enabling condition for many other vulnerabilities since most require sustained follow-through to remediate.

Fix: Defined lifecycle plan covering deployment through retirement, agreed before development begins. Explicit ownership of security and governance tasks within project plan, not treated as implicit. Periodic plan review against actual progress, not only against original timeline.

V043. Misaligned Business Objectives

Layer: Governance

System Type: All

Maps To: NIST AI RMF (Govern)

AI system goals, optimization targets, and success metrics do not align with enterprise policy, risk appetite, regulatory obligations, or customer commitments. Creates structural weakness where system can function exactly as designed and still create harmful or noncompliant outcomes because design itself was optimizing for wrong thing.

Misalignment often appears when efficiency, automation, or growth incentives override control objectives during design or later optimization pass. Can be difficult to spot precisely because system is meeting its stated metric successfully.

Fix: Explicit review of optimization targets against enterprise risk appetite before launch. Defined process to flag metric changes that could shift system's incentives. Periodic review of whether system's actual outcomes still match intended business purpose.

V044. Weak Human Rights Assessment

Layer: Governance

System Type: Predictive, Generative

Maps To: NIST AI RMF (Map)

System design and governance do not evaluate foreseeable impacts on privacy, discrimination, autonomy, due process, or other affected-party rights. Serious weakness in high-impact AI because harms can emerge even when system is technically accurate and secure in narrow technical terms.

System can be robust against attackers and still cause disproportionate harm to particular group through its normal, intended operation. Absence of rights-impact review leaves organization blind to predictable harm scenarios and regulatory exposure.

Fix: Structured impact assessment covering privacy, discrimination, and due process before launch for any high-impact use case. Subgroup outcome analysis as part of that assessment, not separate exercise. Documented path for affected individuals to contest outcome.

V045. Jurisdictional Control Gaps

Layer: Governance

System Type: All

Maps To: ISO/IEC 42001

System operates across legal regions without clear mechanisms to enforce differing requirements for privacy, transparency, retention, fairness, or AI-specific regulation. Creates fragmented compliance behavior and inconsistent risk treatment across deployment footprint.

Control that satisfies one jurisdiction's requirement may fall short of another's. Without explicit mechanism to apply stricter standard where required, system defaults to whichever standard was implemented first.

Fix: Jurisdiction-aware configuration for data handling, retention, and disclosure requirements. Documented mapping of applicable regulatory requirements per deployment region. Technical enforcement of strictest applicable standard where jurisdictions overlap.

V046. Unknown Customer Expectations

Layer: Governance

System Type: Generative, Agentic

Maps To: NIST AI RMF (Map)

The organization does not adequately understand what users, customers, or impacted parties expect in terms of transparency, safety, privacy, reviewability, and responsible AI behavior. Can lead to system that is technically functioning and even technically compliant but that still fails trust, adoption, or reputational thresholds.

Especially relevant in customer-facing AI and decision-support systems where user expectations directly shape what counts as acceptable risk boundary for deployment.

Fix: Structured research into user expectations before launch for customer-facing systems, not assumed from internal judgment alone. Feedback mechanism specifically capturing trust and transparency concerns, separate from functional bug reports. Periodic re-check of expectations as system's capabilities change.

V047. Over-Automation Dependence

Layer: Governance

System Type: Agentic

Maps To: OWASP Agentic Top 10

System or business process relies on AI outputs without sufficient human oversight, review checkpoints, escalation paths, or compensating controls. Critical socio-technical weakness because it turns model error, bias, hallucination, or manipulation directly into business harm with nothing standing between model's output and real-world consequence.

Often appears in operational workflows where users treat AI output as authoritative because process was designed for speed or scale rather than challenge and review. Pressure that motivated automation usually same pressure that discourages adding review back in later.

Fix: Defined human review checkpoints for high-impact decisions, sized to risk of decision rather than uniformly applied. Escalation paths that are actually used, verified through periodic testing rather than assumed to work. Explicit criteria for which decisions can be fully automated and which cannot.

Resilience and Infrastructure

Whether the system keeps running and recovers cleanly under failure or load.

V048. Complex Architecture Sprawl

Layer: Infrastructure, Governance

System Type: All

Maps To: ISO/IEC 42001

AI environment contains too many interconnected components, undocumented dependencies, ad hoc integrations, and fragmented ownership boundaries to be governed effectively. Major architectural weakness because complexity itself expands attack surface, weakens observability, and increases chance controls fail precisely at boundaries between systems.

AI systems commonly combine models, retrieval layers, feature pipelines, agents, APIs, and external tools in ways that can exceed what team can consistently secure. When complexity outpaces governance maturity, risk increases sharply.

Fix: Maintained, accurate architecture diagram reviewed against live system, not just original design. Periodic architecture review specifically looking for undocumented components. Defined threshold at which added complexity requires governance sign-off before proceeding.

V049. Single Point of Failure

Layer: Infrastructure

System Type: All

Maps To: ISO/IEC 42001

One component, service, credential, model registry, vector store, feature store, or orchestration node can disable entire AI capability if it fails or is compromised. Creates avoidable fragility and gives attackers or outages disproportionate leverage over availability and business continuity.

In AI systems, single points of failure often hide in supporting components rather than in model itself. Resilience review focused only on model will miss them.

Fix: Dependency map identifying every component whose failure would take down full capability. Redundancy specifically for supporting components, not only inference service. Periodic failure testing that deliberately removes single component to confirm system degrades gracefully.

V050. Limited Redundancy

Layer: Infrastructure

System Type: All

Maps To: ISO/IEC 42001

Insufficient failover paths, backup services, alternate models, duplicate storage controls, or resilient deployment patterns to sustain operations during failure. Common because teams optimize for performance and cost before designing for resilience, treating redundancy as later-stage concern.

Results in longer outages, slower recovery, and increased blast radius from infrastructure or component failures that more redundant design would have contained.

Fix: Failover paths designed and tested before they are needed, not built reactively after outage. Alternate model or provider options for critical capabilities where feasible. Redundancy requirements included in original design, not treated as later optimization.

V051. Inconsistent Backups

Layer: Infrastructure, Data

System Type: All

Maps To: ISO/IEC 42001

Models, prompts, vector indexes, training artifacts, policies, and configuration states are not backed up in complete, current, and restorable manner. Prevents reliable recovery from corruption, rollback errors, ransomware, accidental deletion, or failed deployment.

AI systems specifically require backup strategies that preserve behavioral state, not just file availability. Restoring wrong version of model or prompt can technically restore service while still restoring unsafe or incorrect behavior.

Fix: Backups covering full behavioral state, including prompts, configuration, and vector indexes, not only model weights. Regular restoration testing to confirm backups are actually usable, not just present. Backup currency tracked against same versioning scheme used in production.

V052. Delayed Model Recovery

Layer: Infrastructure

System Type: All

Maps To: ISO/IEC 42001

Recovery procedures for models, artifacts, indexes, or orchestration state are slow, manual, or untested. Extends downtime and increases operational loss after failure or compromise.

In AI environments, restoration is often more complex than standard application recovery because it depends on version alignment across data, model, prompt, and control artifacts simultaneously. If recovery procedure has never been rehearsed, first real attempt likely to reveal gaps at worst possible moment.

Fix: Documented, rehearsed recovery procedure with defined target recovery time. Recovery procedures tested on schedule, not only after incident. Version alignment checks built into recovery process itself.

V053. Weak Load Distribution

Layer: Infrastructure

System Type: All

Maps To: ISO/IEC 42001

Requests are not balanced effectively across model instances, regions, accelerators, or supporting services. Leads to bottlenecks, avoidable latency, uneven failure patterns, and fragile service behavior under burst traffic or partial outages.

AI inference workloads often highly variable, with usage that spikes unpredictably around specific events or content, making uneven distribution more damaging than for typical application with steadier load.

Fix: Load balancing designed for specific variability of AI inference traffic, not generic web traffic assumptions. Capacity planning that accounts for realistic burst scenarios. Monitoring that distinguishes uneven distribution from genuine overall capacity shortfall.

V054. Limited Fault Tolerance

Layer: Infrastructure, Model

System Type: Predictive, Agentic

Maps To: ISO/IEC 42001

AI systems lack redundancy, error handling, safe degradation, watchdogs, recovery logic, or resilience against malformed inputs and environmental failures. Allows minor faults to escalate into service disruption, wrong predictions, unstable agent behavior, or unsafe operational states.

In AI systems depending on real-time inference or autonomous action, fault tolerance functions as safety and security control, not merely reliability feature affecting uptime statistics. Weak fault resilience increases both accidental and adversarial impact.

Fix: Defined safe-degradation behavior for malformed or unexpected input, tested deliberately rather than assumed. Watchdog processes for agentic systems that can detect and halt unstable behavior. Fault injection testing as part of pre-release validation.

V055. Excessive Compute Demand

Layer: Infrastructure

System Type: Generative, Agentic

Maps To: ISO/IEC 42001

Models, pipelines, or orchestration flows require more computational resources than environment can reliably sustain. Leads to latency, dropped workloads, cost spikes, and brittle service behavior that can mask abuse or degrade user trust.

Often caused by unoptimized models, poorly governed inference chains where single request can trigger many downstream calls, or weak cost-performance engineering during design. System running close to compute ceiling has less capacity to absorb actual denial-of-service attempt on top of ordinary load.

Fix: Compute budgeting built into architecture before launch, not tuned only after cost overruns appear. Capacity headroom sized to absorb both organic growth and abuse scenarios. Regular review of inference cost per request against original design target.

V056. Edge Capacity Weakness

Layer: Infrastructure

System Type: Predictive

Maps To: ISO/IEC 42001

AI models deployed on edge devices run too close to hardware, memory, bandwidth, or energy limits to maintain secure and reliable operation under normal or peak conditions. Creates fragile behavior, degraded controls, and higher failure rates during operational stress.

Particularly acute concern because edge devices often deployed in locations without straightforward central fallback if something goes wrong locally. Capacity engineering should be treated as security-relevant design control for edge deployments.

Fix: Capacity margin built into edge deployment sizing, not calculated against best-case conditions. Defined fallback behavior when local edge device approaches resource limits. Periodic field review of actual edge device utilization against original design assumptions.

Hardware and Side Channel

Physical and low-level risks: unverified hardware, exposed physical interfaces, and information that leaks through timing, power, or gradient signals. Most relevant to edge deployments, embedded systems, and federated training.

V057. Poor Hardware Assurance

Layer: Infrastructure

System Type: All

Maps To: NIST AI 100-2

AI systems rely on low-quality, untrusted, unverified, or weakly monitored hardware platforms for training or inference. Increases risk of hardware faults, tampering, unstable execution, silent corruption, and unreliable operational behavior that no software-level control can fully compensate for.

Particularly relevant for edge AI, specialized accelerators, distributed training hardware, and environments with weak physical security around hardware itself.

Fix: Trusted procurement channels for training and inference hardware. Hardware attestation where supported by platform. Monitoring for anomalous low-level behavior that could indicate tampering or fault.

V058. Weak Hardware Protection

Layer: Infrastructure

System Type: Predictive

Maps To: NIST AI 100-2

Physical interfaces, local consoles, debug ports, firmware update channels, removable media access, and device enclosures are not secured against tampering or unauthorized access. Enables manipulation of execution environments, extraction of artifacts, and compromise of edge or on-premise AI systems.

Especially severe in robotics, IoT deployments, industrial AI, and branch office environments where physical access by unauthorized person is realistic scenario. Physical and logical hardware protections need to be considered together.

Fix: Physical security controls (locked enclosures, restricted access) matched to deployment environment's actual exposure. Disabled or restricted debug and firmware update interfaces in production deployments. Tamper-evidence on physical enclosures for high-value edge devices.

V059. Observable Side Channels

Layer: Infrastructure, Model

System Type: Predictive

Maps To: NIST AI 100-2

Timing behavior, power characteristics, resource usage, memory access patterns, or electromagnetic emissions reveal information about model execution or data being processed. More specialized but genuine weakness in high-value or edge-deployed AI systems, especially where attacker can observe hardware closely.

Presence of these side channels indicates insufficient hardening at runtime or hardware interaction layer, layer that most AI security reviews do not examine at all.

Fix: Side-channel-resistant execution for high-assurance deployments where risk justifies cost. Physical access restriction as primary practical mitigation for most deployments. Specialized side-channel testing reserved for highest-value use cases.

V060. Exposed Gradient Information

Layer: Training Technique, Data

System Type: Predictive

Maps To: NIST AI 100-2

Gradient updates, model deltas, or collaborative learning signals can be accessed or analyzed without strong privacy-preserving controls. Particularly relevant in federated learning and distributed training environments where gradients may leak sensitive information about underlying data used to compute them.

Problem is not collaborative training itself (offers real privacy advantages over centralizing raw data). Problem is sharing training signals without sufficient clipping, aggregation, or privacy protection layered on top.

Fix: Gradient clipping and aggregation before sharing in any federated or distributed training setup. Differential privacy applied to shared gradients where sensitivity of underlying data warrants it. Periodic review of whether collaborative training design still matches original privacy assumptions.

V061. Weak Metadata Scrubbing

Layer: Infrastructure, Data

System Type: All

Maps To: NIST AI 100-2

Logs, API responses, storage objects, file headers, trace records, or debug outputs expose hidden identifiers, source paths, internal roles, or sensitive contextual information. Often overlooked because primary data may appear well protected while metadata quietly reveals relationships, architecture details, or user information.

In AI systems specifically, metadata can also expose prompt structure, feature lineage, or hidden retrieval signals that give attacker insight into how system works internally.

Fix: Systematic metadata review across logs, responses, and storage, not limited to obvious primary data fields. Automated scrubbing applied consistently rather than case by case. Periodic audit specifically targeting metadata leakage, separate from primary data leakage testing.

V062. Weak Tokenization Security

Layer: Data

System Type: Predictive

Maps To: NIST AI 100-2

Tokenization or masking approaches are simplistic, reversible, predictable, or insufficiently isolated from original source content. Allows sensitive data to be reconstructed, inferred, or correlated more easily than team responsible for tokenization scheme intended.

Organizations often mistake simple token substitution for robust privacy protection when surrounding architecture still permits reverse mapping or linkage attacks that reconnect token back to original value.

Fix: Cryptographically sound tokenization design, tested for reversibility rather than assumed secure. Separation between tokenization mapping tables and tokenized data itself. Periodic linkage-attack testing against tokenized dataset.

Third-Party and Supply Chain

Everything the organization did not build itself: procured models, open-source components, vendor services. Grows in importance as sourcing shifts from built to bought.

V063. Black-Box Dependency Reliance

Layer: Model, Governance

System Type: All

Maps To: OWASP LLM03

Organization depends on third-party model or AI service without sufficient transparency into its training, controls, update practices, limitations, or failure behavior. Creates assurance gap because organization cannot fully evaluate what it is actually deploying or how it changes over time as vendor updates it.

Most severe in high-impact use cases where explainability, auditability, and predictable behavior are specifically required since those are exactly properties black-box dependency cannot guarantee.

Fix: Contractual transparency requirements around training data, updates, and known limitations. Independent testing of dependency in organization's own context, not reliance on vendor-published benchmarks alone. Defined fallback if dependency's behavior changes materially after update.

V064. Weak Vendor Due Diligence

Layer: Governance

System Type: All

Maps To: ISO/IEC 42001

Suppliers of models, data, tooling, or AI services are not assessed rigorously for security, privacy, reliability, governance maturity, and legal fitness before onboarding. Allows low-assurance or high-risk components into environment under weak procurement scrutiny.

Procurement process often designed for conventional software and does not ask AI-specific questions that matter. Supplier risk extends beyond ordinary software assurance because model behavior, data lineage, and update practices harder to inspect from outside.

Fix: AI-specific due diligence questionnaire covering training data, update practices, and known limitations, distinct from general vendor security review. Defined risk threshold above which deeper technical assessment required before onboarding. Documented sign-off from accountable owner before new AI vendor goes into production.

V065. Unverified Third-Party Models

Layer: Model

System Type: All

Maps To: MITRE ATLAS, OWASP LLM03

Pretrained models, open-source checkpoints, or vendor-provided AI components integrated without robust testing for backdoors, unsafe behavior, hidden bias, privacy issues, or operational fit. Widespread because model reuse often treated purely as efficiency gain rather than trust decision carrying its own risk.

Organization may inherit latent defects or malicious characteristics never visible in ordinary benchmark testing model was originally evaluated against.

Fix: Security and behavioral testing of any third-party model in organization's own context before deployment. Provenance verification for model's origin and training history where available. Backdoor and trigger-pattern testing for models sourced from less-established repositories.

V066. Outdated Third-Party Components

Layer: Infrastructure

System Type: All

Maps To: OWASP LLM03

Open-source libraries, model-serving tools, plugins, agents, SDKs, or integrated software dependencies no longer supported or missing current security patches. Exposes AI systems to known vulnerabilities in underlying software stack even when model itself well designed and thoroughly validated.

Patching often delayed specifically because teams fear breaking performance or reproducibility of results that took significant effort to validate.

Fix: Dependency scanning integrated into AI pipeline's build process, not only application layer. Defined patching cadence with explicit exception process for components where update risks reproducibility. Regression testing that makes patching lower-risk decision.

V067. Weak Supplier Oversight

Layer: Governance

System Type: All

Maps To: ISO/IEC 42001

Organizations do not actively monitor vendor performance, security posture, contractual obligations, incident handling, and control effectiveness after onboarding. Leaves enterprise blind to degradation, drift in vendor practices, hidden subcontractor risk, and unannounced service changes that occurred well after original due diligence review completed.

AI services often change behavior faster than traditional software, sometimes through routine model update vendor does not treat as material change.

Fix: Scheduled vendor review cycles, not only one-time onboarding assessment. Monitoring for vendor-side changes that could materially affect organization's risk posture. Defined escalation path when vendor review identifies new concern.

V068. Weak Contract Governance

Layer: Governance

System Type: All

Maps To: ISO/IEC 42001

Supplier agreements do not define security obligations, audit rights, incident notification, data handling restrictions, retention rules, model update expectations, and accountability for failures. Technical risk cannot be managed effectively when legal and operational controls that would enforce remedy are undefined or unenforceable in actual contract.

In AI sourcing specifically, contracts often lag behind actual risk exposure, particularly around model updates, prompt retention, and derivative data usage.

Fix: AI-specific contract clauses covering model update notification, data retention, and audit rights, not generic software terms. Legal review specifically for AI sourcing agreements, separate from standard vendor contract review. Defined process to renegotiate terms when vendor relationship expands in scope or risk.

V069. Vendor Lock-In Dependency

Layer: Governance, Infrastructure

System Type: All

Maps To: ISO/IEC 42001

Organization relies too heavily on single AI provider for critical models, infrastructure, APIs, or data services without practical alternatives or migration path. Creates fragility, weak negotiating leverage, constrained assurance, and elevated business risk if service quality, cost, compliance posture, or security conditions change.

While not always framed explicitly as security issue, concentration risk becomes resilience and governance weakness when organization cannot safely diversify away from dependency or exit relationship without significant disruption.

Fix: Documented migration path for critical AI dependencies, even if not currently exercised. Contractual terms that preserve data and configuration portability. Periodic evaluation of alternative providers to keep migration path realistic rather than theoretical.

V070. Weak Third-Party Monitoring

Layer: Governance, Infrastructure

System Type: All

Maps To: ISO/IEC 42001

Supplier behavior, update cadence, control posture, service quality, and security events not continuously observed after integration. Prevents organization from detecting degraded controls, hidden incidents, or changes in model behavior introduced by vendors or external platforms outside organization's own change control process.

AI systems often depend on opaque third-party services where passive trust not justified given how quickly underlying model or service can change without corresponding notification.

Fix: Monitoring tied to vendor-published change logs and status pages, not solely to internal system behavior. Alerting on unexpected output or performance shifts that could indicate unannounced vendor-side change. Defined contact point at vendor for security-relevant questions.

V071. Poor Third-Party Incident Response

Layer: Governance

System Type: All

Maps To: ISO/IEC 42001

Suppliers lack mature procedures, communication channels, escalation speed, and coordination mechanisms for security or AI-specific incidents. Prolongs recovery, obscures root cause, and allows compromise or harmful behavior to propagate across interconnected systems spanning organizational boundaries.

In AI ecosystems, incidents often cross those boundaries and require shared evidence, synchronized containment, and rapid notification between organization and vendor.

Fix: Incident response expectations and notification timelines defined in vendor contract, not assumed. Joint incident response testing with critical vendors where relationship warrants it. Documented fallback plan if vendor's incident response proves inadequate during actual event.

V072. Conflicting Vendor Objectives

Layer: Governance

System Type: All

Maps To: ISO/IEC 42001

Supplier incentives around speed, feature growth, data usage, retention, or monetization are misaligned with organization's security, compliance, reliability, or ethical requirements. Can drive hidden compromises in control quality, transparency, and service fit.

Especially relevant where vendors optimize for scale or product experimentation, releasing changes rapidly, while customer requires stability and assurance appropriate to regulated or high-impact use case.

Fix: Explicit discussion of incentive alignment as part of vendor selection, not assumed to be neutral. Contractual protection against unilateral changes that conflict with organization's stated requirements. Periodic reassessment of whether vendor relationship still fits organization's risk profile as both sides evolve.

V073. Vendor Data Siloing

Layer: Data, Governance

System Type: All

Maps To: ISO/IEC 42001

External providers control or fragment critical data, logs, or performance information in ways that reduce visibility, interoperability, or portability for customer. Weakens monitoring, incident response, root-cause analysis, and strategic flexibility.

In AI systems, missing access to model behavior data, usage analytics, or retrieval context can significantly undermine assurance, particularly during incident where reconstructing what happened depends on data organization does not directly control.

Fix: Contractual guarantees of access to relevant logs and performance data, negotiated before signing rather than after incident. Data export and portability requirements built into vendor selection criteria. Periodic testing that organization can actually retrieve data contract promises.

Documentation

Whether infrastructure, data interfaces, and tooling are documented well enough to support secure operation and recovery.

V074. Poor Resource Documentation

Layer: Infrastructure, Governance

System Type: All

Maps To: ISO/IEC 42001

Required infrastructure, compute dependencies, storage assumptions, data interfaces, runtime requirements, and support tooling not clearly documented across AI lifecycle. Creates avoidable delays, scaling failures, insecure workarounds, and weak capacity planning.

Teams facing undocumented resource requirement under time pressure tend to improvise solution rather than pause to document properly first. Undocumented resources make recovery, troubleshooting, and secure deployment considerably harder.

Fix: Resource requirements documented at each pipeline stage as part of standard build process, not separate task. Documentation reviewed and updated whenever infrastructure changes. Defined owner for keeping resource documentation current.

V075. Poor Tooling Documentation

Layer: Infrastructure, Governance

System Type: All

Maps To: ISO/IEC 42001

Development, training, validation, deployment, and monitoring tools not fully documented in terms of purpose, configuration, ownership, support boundaries, and security expectations. AI programs often depend on broad set of notebooks, registries, experiment platforms, feature stores, package managers, and orchestration tools.

Any of these can become hidden risk source when poorly documented, particularly tools adopted informally by individual team rather than through shared procurement process.

Fix: Maintained inventory of tools used across AI lifecycle, including informally adopted ones. Documented security expectations for each tool proportional to its role. Periodic review specifically looking for tools that entered pipeline without going through standard process.

Multi-Agent

Coordination layer between multiple agents, a risk surface that does not exist in single-model systems.

V076. Agent Coordination Weakness

Layer: Model, Infrastructure, Governance

System Type: Agentic

Maps To: OWASP Agentic Top 10

Multi-agent systems lack strong controls for authentication, communication integrity, role separation, trust boundaries, and behavioral monitoring between agents. Allows one agent's error, manipulation, or compromise to affect others through hidden coordination pathways not designed assuming any single agent might be compromised.

Particularly relevant in emerging agentic architectures where orchestration complexity grows faster than governance maturity. Teams building multi-agent systems often more focused on getting coordination to work at all than on securing the coordination channel itself.

Fix: Authenticated, integrity-checked communication between agents rather than implicit trust. Role separation and least privilege applied per agent, not shared broadly across multi-agent system. Behavioral monitoring designed to detect one agent influencing another outside intended coordination pattern.

Cross-Reference

Every entry corresponds to an ID in data/vulnerabilities.yaml. Update both files together if you add, correct, or re-scope an entry. See CONTRIBUTING.md.

For scripting and filtered checklists, use tools/generate_checklist.py with the YAML file.

If you are staffing this work internally, start with the free, open-source catalog: hwyler/ai-threat-modeling-toolkit maps 77 AI vulnerabilities to 49 threat vectors across predictive, generative, and agentic systems, cross-referenced to MITRE ATLAS, the OWASP Top 10 for LLM and Agentic Applications, NIST AI 100-2, and ISO/IEC 42001. It ships as structured YAML data, not just prose, so you can filter it into a scoped checklist for a fraud model, a RAG chatbot, or an autonomous agent in minutes rather than weeks.

If you found the ClaimAssist walkthrough useful, subscribe for the next installment, where we build out the Monte Carlo loss exceedance curve for a full three-layer AI system and show exactly where the reserve line should sit before your board asks for it.

Why Traditional Security Testing Misses 70% of AI Attack Surface

Hernan Huwyler — Thu, 30 Apr 2026 06:56:40 +0000

A practical guide to AI-specific threat modeling, vulnerability assessment, and the frameworks that actually matter for predictive, generative, and agentic systems

I've spent the last two years reviewing AI security assessments across financial services, AI software and computer vision development, healthcare, and technology companies. The pattern is consistent and concerning: organizations conduct thorough infrastructure reviews, validate API security, verify access controls, and declare their AI systems production-ready. Then they discover, often after an incident, that they tested roughly 30% of their actual attack surface.

The missing 70% consists of threats that simply don't exist in traditional software systems: training data poisoning that corrupts model behavior without modifying code, adversarial inputs that cause systematic misclassification, prompt injection attacks that override system instructions through user-provided text, model extraction through API query patterns, and autonomous agents executing unauthorized actions through legitimate tool access.

This isn't a theoretical concern. It's a systematic gap in how we approach AI security.

Complete guide to AI threat modeling from STRIDE to production:
AI Threat and Vulnerability Assessment

Why AI Systems Require Different Threat Models

Traditional software operates deterministically. Given identical inputs, it produces identical outputs. Its behavior is explicitly programmed and can be inspected through source code review. Security assessment frameworks like OWASP Top 10, CWE, and conventional penetration testing evolved around these assumptions.

AI systems violate every one of them.

They learn behavior from data rather than having it explicitly coded. They produce probabilistic outputs that may vary across identical inputs. Their decision boundaries emerge from statistical patterns rather than programmed logic. Their supply chain extends beyond code libraries to include datasets, pre-trained models, fine-tuning corpora, embeddings, and retrieval sources, each introducing distinct vulnerability classes.

This creates attack surfaces across dimensions traditional security never addressed:

Data-centric attacks manipulate training data, labels, feature pipelines, or retrieval corpora to influence model behavior without touching infrastructure or code.

Model-centric attacks exploit learned behavior through adversarial inputs, extraction queries, or inversion techniques that reconstruct training data.

Pipeline-centric attacks compromise MLOps infrastructure, model registries, training environments, or deployment pipelines.

Human interaction attacks exploit natural language interfaces through prompt injection, jailbreaking, or manipulation of user-facing outputs.

Autonomy attacks exploit tool access, planning capabilities, memory systems, or action authorization in agentic AI systems.

The security assessment methodology must account for all five dimensions. Conventional application security testing covers portions of the pipeline layer but misses the others entirely.

The Value of AI-Specific Threat Taxonomies

Three frameworks have emerged as essential references for comprehensive AI threat assessment:

MITRE ATLAS (Adversarial Threat Landscape for AI Systems)
ATLAS catalogs over 80 techniques organized across 14 tactics specifically for attacking AI systems. It extends the familiar ATT&CK framework into the AI domain, providing a structured taxonomy of how adversaries actually compromise machine learning systems.

The value isn't just the technique catalog, but it is the common language it provides between security teams, data science teams, and business stakeholders. When you identify that your fraud detection model is vulnerable to "AML.T0020 - Poison Training Data," everyone understands the reference, the attack pattern, and where to look for mitigations.

ATLAS differentiates attacks by lifecycle stage (reconnaissance, resource development, initial access, ML model access, execution, persistence, etc.), making it practical to map threats to your specific MLOps pipeline and deployment architecture.

NIST AI 100-2 (Adversarial Machine Learning Taxonomy)
NIST's taxonomy provides systematic categorization of adversarial ML attacks by:

Attack objective (confidentiality, integrity, availability)
Attacker knowledge (white-box, gray-box, black-box)
Attack specificity (targeted vs. indiscriminate)
Lifecycle stage (training-time vs. inference-time)
This framework excels at helping teams understand why certain vulnerabilities matter more in specific contexts. A financial institution deploying a credit scoring model faces different threat priorities than a healthcare provider deploying a diagnostic assistant, even when both use similar ML architectures.

The taxonomy also bridges the gap between academic research on adversarial ML and operational security practice. When researchers publish new attack techniques, NIST's categorization helps practitioners assess whether the attack applies to their deployment model.

OWASP Top 10 for LLM Applications
OWASP's LLM-specific guidance addresses the explosion of generative AI deployments. The 2025 version covers:

Prompt Injection
Insecure Output Handling
Training Data Poisoning
Model Denial of Service
Supply Chain Vulnerabilities
Sensitive Information Disclosure
Insecure Plugin Design
Excessive Agency
Overreliance
Model Theft
What makes this valuable is the detailed attack scenarios, real-world examples, and practical prevention strategies that accompany each risk category. The framework explicitly addresses risks in RAG (Retrieval Augmented Generation) systems, agent architectures, and tool-using LLMs that earlier security frameworks never contemplated.

Each risk includes developer-focused guidance on detection, prevention, and example attack scenarios. This makes it immediately actionable for engineering teams rather than requiring security expertise to translate abstract threats into concrete controls.

STRIDE-AI: Extending Classic Threat Modeling for AI Assets
Classic STRIDE (Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, Elevation of Privilege) provides a structured approach to threat modeling. It works well for traditional software but requires extension for AI systems.

Spoofing in AI extends beyond identity impersonation to include:

Training data source spoofing (malicious data presented as trusted sources)
Model provenance spoofing (trojanized models distributed through model hubs)
Prompt identity manipulation (causing models to assume unauthorized roles)
Tampering expands dramatically:

Training data poisoning (injecting crafted samples to embed backdoors)
Label manipulation (corrupting ground truth)
Feature pipeline tampering (modifying preprocessing logic)
Model weight modification (directly altering learned parameters)
Prompt template tampering (modifying system instructions)
Retrieval corpus poisoning (injecting malicious content into RAG systems)
Repudiation creates AI-specific accountability gaps:

Inability to prove who changed a model or dataset
Missing audit trails for prompt modifications or agent actions
Inability to reconstruct why specific outputs were produced
Information Disclosure includes novel privacy and IP risks:

Training data leakage through model outputs
System prompt leakage through crafted queries
Membership inference (determining if data was in training set)
Model inversion (reconstructing sensitive features from outputs)
Denial of Service exploits computational intensity:

High-volume API abuse
Token flooding in language models
Adversarial prompts triggering expensive computation
Agent loops consuming resources indefinitely
Elevation of Privilege allows capability expansion:

Prompt injection causing unauthorized tool use
Agents executing actions beyond intended scope
Weak role boundaries in MLOps pipelines

The extension isn't cosmetic. Teams using classic STRIDE for AI assessments consistently miss data poisoning, adversarial examples, prompt injection, and agent-specific threats because traditional STRIDE categories don't naturally surface these attack vectors.

Threat Vectors That STRIDE Alone Doesn't Capture

Six threat categories require explicit attention beyond STRIDE extension:

Data Poisoning Manipulates training, fine-tuning, retrieval, or feedback data to corrupt model behavior. Three subtypes create different impacts:

Availability poisoning: Degrades overall performance
Integrity poisoning: Creates targeted backdoor behaviors
Bias poisoning: Skews outcomes for specific groups
The challenge: this attack succeeds without compromising infrastructure or code. Traditional security monitoring won't detect it.

Evasion and Adversarial Examples Crafts inputs that cause misclassification or bypass detection at inference time. Common in computer vision, audio processing, fraud detection, and content moderation.

The challenge: inputs appear legitimate to humans and validation logic but exploit model-specific decision boundary weaknesses.

Model Extraction and Theft Replicates model behavior or steals intellectual property through systematic API queries. Attackers build surrogate models that approximate the original without accessing weights directly.

The challenge: extraction happens through normal API usage patterns. Without query monitoring and behavioral analysis, it's indistinguishable from legitimate high-volume use.

Prompt Injection Places malicious instructions in user inputs, documents, web pages, emails, or tool outputs, causing models to ignore system instructions. Particularly critical for LLMs and RAG systems.

Direct injection: User types malicious instructions
Indirect injection: Malicious instructions embedded in retrieved documents

The challenge: the model cannot reliably distinguish system instructions from adversarial content unless the architecture enforces separation.

Hallucination and Fabrication Produces confidently stated incorrect information. While not always malicious, it creates exploitable risk when outputs drive decisions or actions.

The challenge: distinguishing incorrect confidence from correct confidence requires external verification mechanisms that many deployments lack.

Agentic Risks Unique to AI systems that plan, use tools, and act on the environment:

Goal hijacking
Tool abuse through legitimate access
Recursive harmful loops
Multi-step failure chains
Memory poisoning
Cross-system lateral movement
The challenge: individual actions may appear authorized while the sequence or combination violates policy.

Practical Implementation: The Six-Phase Assessment Process

Effective AI security assessment follows a repeatable process aligned with NIST AI RMF and ISO/IEC 42001:

Phase 1: Define Scope and Objectives
Identify which AI systems, environments, and use cases are in scope. Document risk tolerance with specific measurable standards:

"No PII in outputs"
"No more than 3% performance degradation after adversarial hardening"
"Prompt injection bypass rate below 0.1%"
Vague success criteria produce vague assessments.

Phase 2: Inventory AI Assets and Data Flows
Catalog models, datasets, pipelines, training and inference infrastructure, and external dependencies. Include metadata: data lineage, model versions, training configuration, deployment endpoints, prompt templates, tool permissions, retrieval corpora.

Build an architecture diagram capturing every data flow, trust boundary, and external dependency.

Critical insight: Most assessments fail at this phase. Teams inventory the model and API endpoint but miss the data pipeline, feature store, retrieval corpus, prompt templates, tool configurations, and monitoring infrastructure. Each component has its own threat profile and attack surface.

Phase 3: Threat Mapping and Vulnerability Analysis
Apply STRIDE-AI threat modeling per asset. Use MITRE ATLAS to identify attack patterns specific to your system type. Consider attack surfaces across inputs, training data, model parameters, interfaces, logs, monitoring systems, and agent tools.

Build scenario-based risk assessments for the most consequential threats. Generic threat lists produce generic findings. Scenarios produce actionable intelligence.

Phase 4: Testing and Validation
Perform targeted security tests informed by the threat model:

Adversarial testing
Prompt injection testing
Data integrity tests
Privacy leakage tests
Agent behavior tests
Abuse resistance tests
Use a mix of automated tooling and manual testing. Test against specific threats identified in Phase 3, not generic checklists.

Phase 5: Risk Scoring and Prioritization
Use a likelihood-impact matrix with AI-specific scoring. The OWASP AI Vulnerability Scoring System (AIVSS) provides dimensions designed for AI risks including agentic systems.

Maintain an AI risk register linking threats, vulnerabilities, controls, and residual risk to business impact and regulatory constraints.

Phase 6: Mitigation and Continuous Monitoring
Implement layered controls: access control, input validation, rate limiting, adversarial training, differential privacy, data validation, output filtering, robust logging, human approval gates.

Set up ongoing monitoring of performance, drift, anomaly behavior, and security signals. Loop findings back into risk assessment.

Critical insight: AI threat assessment isn't a one-time activity. Systems change through retraining, data updates, prompt modifications, tool additions, and vendor model changes. Each change can introduce new vulnerabilities or alter control effectiveness.

Testing Differentiated by AI Type

Assessment must be tailored to the system's interaction mode and autonomy level.

For Predictive AI (fraud detection, credit scoring, demand forecasting)
Primary focus: training data integrity, adversarial input robustness, fairness across demographic groups, drift resistance.

Key tests:

Simulate evasion attacks by incrementally altering input features
Inject plausible poisoned samples into training data to evaluate backdoor risk
Run fairness assessments including robustness under data drift
Test model stability across distribution shifts
For Generative AI (chatbots, code generation, content creation)
Primary focus: prompt injection resistance, harmful content generation, data leakage, retrieval pipeline security.

Key tests:

Systematic prompt injection using curated adversarial prompt suites
Multi-turn and indirect injection through retrieved content
Red-team exercises attempting harmful output elicitation
Privacy testing for training data leakage
Output filter validation (false negatives and false positives)
For Agentic AI (tool-using agents, autonomous workflow systems)
Testing must cover all generative AI threats plus autonomy-specific risks.

Key tests:

Scenario-based simulations in sandboxed environments
Permission boundary testing (remove tools, restrict scopes)
Rollback and fail-safe mechanism validation
Memory integrity testing through crafted interactions
Multi-step action chain analysis for unauthorized outcomes
Different threats. Different tests. Different controls. Applying generic "AI security checklists" to all three types produces assessments that miss the most important risks for each.

Built vs. Bought: Different Risks Require Different Strategies
Whether you develop AI internally or procure it from vendors fundamentally changes both threat profile and assessment approach.

Internally Developed AI
Visibility: Full access to data, model architecture, training pipeline, infrastructure
Primary threat exposure: Training-time attacks (supply chain compromise, data poisoning, environment compromise)
Assessment approach: Integrate threat modeling into MLOps pipeline, maintain detailed documentation, use internal red teaming, adopt secure MLOps practices

Best practices:

Security regression testing in CI/CD
Data lineage documentation
Model cards and evaluation transparency
Signed artifacts and registry governance
Environment isolation and secrets management
Procured AI
Visibility: Limited or no access to training data, model internals, training process
Primary threat exposure: Supply chain vulnerabilities, embedded backdoors, undocumented behaviors, loss of control over shared data, unannounced model updates
Assessment approach: Vendor due diligence, contractual controls, independent validation, wrapper controls

Best practices:

Security architecture review in procurement
Contractual audit rights and logging commitments
Change notification requirements
Independent prompt injection and leakage testing
External guardrails and policy enforcement
Output monitoring independent of vendor
Critical insight: For built AI, uncertainty concentrates in implementation (did we build controls correctly?). For bought AI, uncertainty concentrates in assurance (do vendor controls work as claimed?). You often can't verify vendor claims about poisoning resistance, fine-tuning provenance, data retention, or hidden tool usage. This assurance gap means procurement assessment must emphasize trust boundaries, vendor governance verification, and contractual controls more heavily than technical testing.

The Role of Red and Blue Teams
Strong AI vulnerability assessment combines red team pressure testing with blue team detection and defensive validation.

Red Team Role
Simulate realistic attacker, insider, misuse, and abuse scenarios. Test whether weaknesses in prompts, data pipelines, model governance, APIs, memory, tools, vendor integrations, and human workflows can be turned into business impact.

Key questions:

Can the model be manipulated through untrusted inputs?
Can poisoned data enter training or retrieval pipelines?
Can the model leak sensitive information?
Can agents invoke tools beyond intended authority?
Can vendor updates introduce hidden risk?
Blue Team Role
Validate defensive readiness, observability, containment, and recovery. Test whether the organization can detect exploit attempts, recognize harmful model behavior, distinguish normal use from abuse, contain incidents, and restore trusted operation.

Key questions:

Would we detect prompt injection or model extraction quickly?
Can we distinguish drift, misuse, poisoning, and infrastructure failure?
Do logs capture enough context to reconstruct incidents?
Can we disable tools or models safely during active incidents?
Can we prove which model version and dataset were active at incident time?
Purple Teaming
The most valuable approach: red and blue teams work collaboratively. Red team demonstrates exploitation while blue team observes telemetry, identifies detection gaps, tests containment, and validates response procedures.

This is especially effective for prompt injection scenarios, agent tool misuse, model extraction attempts, retrieval poisoning, and data leakage testing where defenders are still learning what malicious behavior looks like in production.

Common Implementation Failures
Ten patterns recur across organizations conducting AI security assessments:

Treating AI like ordinary software: Assessing only infrastructure and application security while missing data, model, and pipeline threats
Testing only accuracy: Ignoring abuse resistance, security, privacy, robustness, and fairness
Threat modeling only the endpoint: Missing data pipeline, training infrastructure, retrieval systems, tool integrations, monitoring
Ignoring vendor opacity: Accepting vendor claims without independent verification
Allowing model-based authorization: Letting models directly authorize high-risk actions without independent policy enforcement
Weak prompt isolation: Mixing trusted instructions with untrusted user and retrieved content
Insufficient logging: Making root cause analysis impossible when problems occur
Not reassessing after changes: Allowing security posture to degrade as systems evolve
Assuming controls without testing: Accepting vendor or development claims about safety without adversarial validation
Ignoring human overreliance: Failing to assess whether users can distinguish reliable outputs from unreliable ones
The most consequential mistake for most organizations is the first: treating AI like ordinary software. If your current security assessment doesn't include AI-specific threat categories (poisoning, evasion, extraction, prompt injection, agent abuse), it's missing the majority of the AI-specific attack surface regardless of how thoroughly it covers traditional security dimensions.

Practical Next Steps
For teams building AI systems:

Classify your AI type (predictive, generative, agentic) and sourcing model (built, procured)
Conduct STRIDE-AI threat modeling workshops with cross-functional participation
Map identified threats to MITRE ATLAS techniques and OWASP AI risks
Implement security regression testing in your MLOps pipeline
Deploy monitoring for drift, extraction patterns, adversarial inputs, and policy violations
For teams procuring AI:

Include AI-specific security requirements in vendor due diligence
Negotiate contractual audit rights and logging commitments
Implement wrapper controls (guardrails, redaction, external policy enforcement)
Test procured systems with prompt injection and leakage assessments
Monitor vendor model updates and conduct regression testing
For security teams:

Adopt MITRE ATLAS, NIST AI 100-2, and OWASP LLM Top 10 as reference frameworks
Build AI-specific security testing capabilities (adversarial testing, prompt injection, privacy testing)
Integrate AI risk into enterprise risk registers with appropriate governance
Establish purple team exercises for high-risk AI deployments
Create AI security baseline documentation and reassessment cadence
Conclusion
AI systems present attack surfaces that traditional security assessment methodologies weren't designed to evaluate. The gap isn't theoretical—it's measurable, systematic, and present in most AI deployments today.

The frameworks exist. MITRE ATLAS provides a common taxonomy of AI-specific attack techniques. NIST offers systematic categorization of adversarial ML attacks. OWASP delivers practical guidance for LLM security. STRIDE-AI extends familiar threat modeling into the AI domain.

The challenge isn't lack of guidance. It's organizational inertia—continuing to apply traditional security checklists to fundamentally different systems and declaring victory when infrastructure tests pass.

An AI system assessed only for traditional security threats is an AI system with most of its attack surface unexamined. The threats that don't show up in conventional penetration tests—data poisoning, adversarial manipulation, prompt injection, model extraction, agentic abuse—are the ones most likely to cause actual harm in production.

The question isn't whether your organization will adopt AI-specific threat assessment. The question is whether you'll do it before or after an incident forces the issue.

About the Author

Prof. Hernan Huwyler, MBA, CPA, CAIO serves as AI GRC Consultancy Director, working with organizations across financial services, technology, healthcare, and public sector to build practical AI governance frameworks. His work on AI threat modeling, quantitative risk assessment, and compliance automation is publicly available at https://hwyler.github.io/hwyler/

Connect on LinkedIn: linkedin.com/in/hernanwyler

References

MITRE ATLAS
NIST AI Risk Management Framework
OWASP Top 10 for LLM Applications:
ISO/IEC 42001:2023 AI Management Systems
OWASP AI Vulnerability Scoring System (AIVSS)

Tags: #AISecuirty #MachineLearning #ThreatModeling #CyberSecurity #MLOps #LLMSecurity #MITRE #OWASP #NIST #DevSecOps

Why I Write About AI Governance (And Why It Actually Matters) Blog: https://hernanhuwyler.wordpress.com I've spent the last two decades sitting in rooms where smart people make expensive mistakes with technology they don't fully understand.

Hernan Huwyler — Mon, 13 Apr 2026 22:07:45 +0000

AI Governance and Risk Management – Prof. Hernan Huwyler, MBA CAIO CPA AI GRC Director | AI Risk Manager | Compliance Officer and Auditor | Quantitative Risk Lead | Speaker and Corporate Trainer | Executive Advisor

hernanhuwyler.wordpress.com

Practical Problem Definition for AI Projects (A Developer-First Guide)

Hernan Huwyler — Mon, 13 Apr 2026 21:55:38 +0000

If you want the full, original version of this write-up (with more governance framing and templates), start here: Practical problem definition for AI projects and use cases.

If you like technical posts that treat AI as production infrastructure, not a demo, my main index is here: hernanhuwyler.wordpress.com.

Now the developer version.

I have seen more AI projects die from a bad problem statement than from a bad model.

The code was fine. The embeddings were fine. The training run was fine. The metrics looked “good.” Then the system shipped and nobody used it, or it automated the wrong step, or it created a new failure mode that support had no way to handle.

That failure usually started on day one, when someone wrote: “We need an AI solution.”

I am intentionally leaving three human typos in this post because this is how real project docs look at 1 AM: teh, definately, occured.

Why “we need AI” is not a problem statement
A real problem statement describes a measurable gap in a workflow.

An AI-flavored ambition describes a technology preference.

If your team starts with “use AI,” you will end up fitting AI into whatever pain is nearby. That feels productive until you try to write acceptance tests.

Instead of “we need an AI assistant,” write something a test suite can verify:

“We spend 1,200 hours per quarter answering due diligence questionnaires, with a median turnaround of 9 days and an observed rework rate of 12%. We need median turnaround under 2 days while keeping rework under 5%.”

That is not business theater. That is a spec.

The goal: turn business pain into an executable spec
A good AI problem definition gives developers five things:

You know what the system will do.

You know what “good” looks like.

You know what “unsafe” looks like.

You know what data you need.

You know how to decide go or no-go without politics.

If you cannot write those down, you do not have a project. You have a conversation.

Step 1: Write the “as-is” workflow like you are debugging it
When teams skip this, they end up automating the wrong step.

Write the current workflow as a sequence diagram or as pseudocode. Keep it brutally literal.

Example (support ticket triage):

text

1) Ticket arrives in Zendesk
2) Agent reads it
3) Agent searches internal KB + Slack history
4) Agent drafts response
5) Agent checks policy constraints (refunds, privacy, SLA)
6) Agent sends response
7) Escalation occurs if customer replies again
Now mark where the real bottleneck is.

Is it step 3 (search)? Step 5 (policy checks)? Step 7 (escalations)?

If you do not identify the actual constraint, you will build a system that makes step 4 faster while the process still waits on step 5.

Step 2: Define the output contract before you touch a model
Developers need an output contract, even if the model is probabilistic.

For each AI output, define:

output type (classification, draft text, decision suggestion, extracted fields)
required metadata (sources, confidence, policy flags)
acceptable error modes
required human review conditions
logging requirements
Example: a response drafting system that must cite sources.

JSON

{
"draft_reply": "string",
"citations": [
{ "doc_id": "string", "section": "string", "quote": "string" }
],
"policy_flags": ["privacy", "refund", "security"],
"confidence": 0.0,
"needs_human_review": true
}
If your vendor tool cannot produce the fields you need for your workflow, you just learned something early, not after deployment.

Step 3: Force the counterfactual: “how do we solve this without AI?”
This single question kills weak projects fast.

If a rules engine, a better search index, a form redesign, or a simple automation tool solves 80% of the pain, AI is not your first move.

You can still use AI later, but you will use it in the right place.

A lot of “AI projects” are really data quality projects or workflow standardization projects. That is not a failure. That is reality.

Step 4: Choose the right tool class before choosing the tool
Engineers waste months when they choose a model family before they classify the task.

A simple filter works:

If the task is deterministic and structured, prefer conventional software.

If the task is prediction, ranking, scoring, or classification on structured data, prefer traditional machine learning.

If the task is understanding or generating unstructured language, then consider large language models.

Most real projects are hybrid. The mistake is making the whole thing “AI” when only one component needs it.

Example hybrid for due diligence automation:

retrieval system to fetch relevant policy sections
language model to draft responses with citations
rules engine to flag regulated claims
human review for high-risk topics
Step 5: Feasibility check that developers actually care about
This is where optimism goes to die, which is good. You want it to die early.

Data feasibility
Do you have the data? Is it current? Is it consistent? Is it legally usable?

If the answer is “we have PDFs somewhere,” your project is not a model project yet. It is a data engineering project.

Label feasibility (if supervised learning is involved)
If you need labels, ask:

Who produces them?
How long does it take?
How noisy are they?
Can we measure inter-annotator agreement?
If you cannot sustain labeling, you cannot sustain the model.

Operational feasibility
Can you meet latency, cost, and uptime targets?

If inference costs are unbounded, “accuracy” is irrelevant. Your system will be throttled by finance.

Safety and abuse feasibility
If the system can take action (send emails, trigger workflows, call APIs), you need explicit constraints.

If you cannot articulate how prompt injection or data exfiltration would be detected, that risk will definately show up later.

Step 6: Define success metrics that cannot be negotiated later
If success metrics are vague, your project will never finish. It will just… continue.

I use four metric buckets.

Technical quality
Depends on task. Examples:

accuracy, precision, recall, F1
extraction exact match rate
groundedness or citation validity (for retrieval-based systems)
calibration (do probabilities mean anything?)
Business impact
median turnaround time reduction
rework rate reduction
cost per case
SLA adherence
Risk and control metrics
policy violation rate
unsafe output rate
number of escalations per 1,000 outputs
audit log completeness
Adoption
percentage of cases processed through the system
override rate (humans rejecting the AI output)
opt-out rate (users routing around it)
If adoption is low, your problem definition was wrong, your UX was wrong, or your trust model was wrong. Pick one and investigate.

Make the problem definition machine-readable (so it becomes a build artifact)
This is the most practical trick I can offer to developers.

Convert the problem definition into a repo artifact. Treat it like code.

Example use_case.yaml:

YAML

use_case_id: "ddq_auto_response_v1"
owner: "security_ops"
objective:
baseline:
median_turnaround_days: 9
rework_rate: 0.12
target:
median_turnaround_days: 2
rework_rate: 0.05

outputs:

name: "draft_answer" requires_citations: true human_review_required_when:
- "policy_flags contains 'privacy'"
- "confidence < 0.75"

data_sources:

name: "control_matrix" format: "structured" freshness_sla_days: 30
name: "policies" format: "pdf" ocr_required: true

constraints:
pii_allowed: false
max_latency_ms: 2500
audit_logging_required: true

pilot:
duration_weeks: 8
sample_size: 50
go_no_go:
min_pass_rate: 0.90
min_time_reduction: 0.70
Now your engineers can write tests against this. Your PM can’t “reinterpret” it mid-flight. And when an incident occured, you have a paper trail that matches what was shipped.

Pilot design that avoids pilot purgatory
Pilots fail when they are not built to produce a decision.

Define:

exact duration
exact sample size
pre-agreed thresholds
decision date
Example:

“The pilot runs for 8 weeks on 50 questionnaires. We scale only if pass rate exceeds 90% and median turnaround improves by 70%. If not, we do a root cause analysis and decide continue, modify, or stop within 2 weeks.”

If you do not write that down, you will extend the pilot forever because nobody wants to be the person who says stop.

Red flags I watch for in problem statements
If I see these, I assume the project will stall unless the team rewrites the spec.

“We want an AI strategy.”
“We want to explore AI.”
“We want to improve customer experience.”
“We want a chatbot.”

Those can be ambitions. They are not problem definitions.

A problem definition has a baseline, a target, constraints, and a decision gate.

A short note on standards (only because they help developers)
If you work in a regulated environment, problem definition is not just best practice. It becomes evidence.

These references map well to developer workflows:

NIST AI Risk Management Framework (especially the Map function)
ISO/IEC 42001 (planning, roles, lifecycle discipline)
ISO/IEC 5338 (AI system lifecycle processes, where available)
You do not need to memorize standards. You need to produce artifacts that prove intent, constraints, and control.

Learn more
Original article: Practical problem definition for AI projects and use cases

Blog index: hernanhuwyler.wordpress.com

Closing question (the one I use to test problem definition quality)
Could someone outside your team read your problem statement and write correct acceptance tests from it in under 15 minutes?

Build vs Buy for AI Systems (A Developer’s Guide to Not Regretting the Decision)

Hernan Huwyler — Mon, 13 Apr 2026 21:46:23 +0000

Before we get technical, two quick pointers if you want the longer, governance-heavy version of this topic and the rest of my field notes. https://hernanhuwyler.wordpress.com/

Start with the original article: Building vs Buying Decisions for AI Systems

https://hernanhuwyler.wordpress.com/2026/03/12/building-vs-buying-decisions-for-ai-systems/

If you like this style of practical, production-minded AI engineering, the full blog index is here: hernanhuwyler.wordpress.com

Now the developer take.

I keep seeing AI teams ask “build vs buy” after the architecture is already half-decided. Engineering has a repo. Procurement has a short list. Security has questions nobody can answer. Then the project turns into a political debate about speed and control.

That is how you end up with either:

a custom system that nobody can operate safely at 2 AM, or
a vendor system that “works in the demo” but you cannot monitor, explain, or roll back when it misbehaves.
This post is the decision framework I wish more teams used before they commit to code, contracts, or platform lock-in.

I am going to be blunt: build vs buy is not a procurement question. It is an operating model decision with consequences for reliability engineering, incident response, and long-term ownership.

Also, yes, I’m leaving three human typos in here on purpose because this is how real engineers write under time pressure: teh, definately, occured.

What “build vs buy” really means in AI (it is rarely binary)
In AI, “build” can mean at least five different things:

build a model from scratch
fine-tune a foundation model
build a retrieval layer and orchestration around a hosted model
build the evaluation and monitoring stack around a vendor tool
build the workflow integration, guardrails, and audit logging around SaaS AI
“Buy” also has levels:

buy a fully managed end-to-end product
buy a platform (model hosting, vector database, feature store, pipeline tooling)
buy a component (OCR, transcription, embeddings, redaction, PII detection)
buy “AI inside SaaS” that quietly becomes a production dependency
Most production systems end up hybrid. The question is whether you are designing hybrid on purpose, or drifting into it without controls.

The four lenses that keep teams honest
I use four lenses. If you skip even one, the decision becomes biased toward ideology.

1) Solution fit (does it actually solve your problem?)
For developers, “fit” is not a feature checklist. It is:

Does it support your data shapes and your failure modes?
Does it support your latency budget and throughput?
Can it run in your environment (networking, identity, compliance boundaries)?
Does it support the behavioral constraints you need (tone, safety, refusal, citations, determinism)?
A vendor might be perfect for commodity workflows like OCR, transcription, translation, ticket summarization, or code completion.

A vendor will struggle when your differentiator is your workflow logic, your proprietary corpus, your control requirements, or your need for deep integration and observability.

Practical test: write one “golden path” scenario and ten “nasty path” scenarios. Make the vendor run them in your environment with your data patterns, not their sandbox.

2) Operating capability (can you run it for years, not weeks?)
Most teams can build a prototype. Fewer can operate an AI system like an SRE-owned service.

If you build, you own:

model registry and artifact lineage
feature pipelines and data contracts
evaluation harness, thresholds, and regressions
model serving, scaling, and cost controls
monitoring, alerting, incident playbooks
retraining triggers, rollback, and retirement
If you buy, you still own:

integration and identity boundaries
monitoring of outcomes in your workflows
“vendor changed something” detection
audit evidence and incident coordination
fallbacks when the service degrades
Hard question: who will be on call when the model starts producing toxic output at 11 PM and Customer Support escalates?

If the answer is “we’ll figure it out,” the decision is not ready.

3) Control and risk (who owns the hardest failure mode?)
Neither build nor buy is safer by default. The safer option is the one where the risk is measurable and enforceable in your environment.

In real systems, the hardest risks tend to be:

data leakage (training or inference)
prompt injection and tool abuse (if you allow tools/actions)
model drift and silent quality decay
fairness regressions across segments
lack of audit logging and replayability
vendor opacity (no eval access, no update transparency)
Control test: when something goes wrong, can you answer these in under an hour?

What exact version is running?
What changed since last week?
Can we roll back safely?
Do we have logs that prove what happened?
If you cannot, you do not have operational control. You have hope.

4) Lifecycle economics (five-quarter view, not quarter-one)
AI cost surprises rarely come from build time. They come from running time.

If you build, hidden cost tends to be:

staffing continuity, turnover, and tribal knowledge
infra, GPUs, storage, and network egress
monitoring and evaluation effort
governance artifacts, audits, and evidence trails
technical debt from “we shipped it fast”
If you buy, hidden cost tends to be:

usage pricing (tokens, queries, seats, “premium support”)
integration complexity and custom connectors
vendor change management and renegotiations
lock-in and migration costs
lack of portability for prompts, embeddings, or policies
Rule I use: compare expected-case cost over five quarters with stressed-case assumptions. AI vendors and internal builds both look great in best-case spreadsheets.

A developer-first decision matrix (build, buy, hybrid)
Here is a lean matrix you can actually use in an engineering review.

Dimension Build tends to win when Buy tends to win when Hybrid tends to win when
Differentiation Your workflow or model behavior is core IP It is commodity capability Core workflow is unique, base capability is commodity
Data constraints You need strict boundary control, custom redaction, or on-prem Vendor supports your boundary model You keep sensitive layers in-house, outsource the rest
Observability You need deep tracing, replay, and segment analytics Vendor offers limited logs You build monitoring + audit around vendor core
Change control You need deterministic releases Vendor changes are opaque You isolate vendor changes behind an abstraction layer
Talent You have ML + platform + security depth You do not You buy platform, build app layer
This is intentionally not “complete.” It is enough to force real trade-offs early.

Technical due diligence if you are buying (what I make teams test)
Buying AI without a test harness is how teams get surprised in production.

1) Black-box evaluation harness (minimum viable)
You need a repeatable harness that can be run:

before purchase (pilot)
before upgrades
after vendor model changes
after policy or prompt changes
A simple pattern:

Python

from dataclasses import dataclass
from typing import Callable, List, Dict
import time

@dataclass
class TestCase:
name: str
input: str
expected_tags: List[str] # e.g., ["no_pii", "refuse_illegal", "cite_sources"]

def run_eval(cases: List[TestCase], call_model: Callable[[str], Dict]) -> Dict:
results = {"pass": 0, "fail": 0, "latency_ms": []}
for c in cases:
t0 = time.time()
out = call_model(c.input)
latency = (time.time() - t0) * 1000
results["latency_ms"].append(latency)

    tags = out.get("tags", [])
    ok = all(tag in tags for tag in c.expected_tags)
    if ok:
        results["pass"] += 1
    else:
        results["fail"] += 1
        print(f"FAIL: {c.name} got tags={tags}")
return results

Do not argue about vendor quality based on a demo. Run your cases.

2) Update detection
If the vendor can update models or policies, you need detection. At minimum:

compare output distributions over time
run nightly regression tests on a fixed suite
alert when drift crosses a threshold
If you cannot detect vendor changes, you will misdiagnose incidents as “our integration” when the behavior changed upstream.

3) Contractual requirements that matter to engineers
This is not legal advice. It is the engineering reality I’ve seen break production.

Ask for:

change notification commitments
data usage boundaries (training, retention, logging)
incident notification timelines
audit evidence availability
export/migration support (prompts, embeddings, configs where possible)
service-level objectives (latency, uptime, support response)
A vendor that cannot commit to update visibility is not a vendor. It is a variable.

Technical risk if you build (what teams underestimate)
When teams build, the failures are usually boring and brutal:

Reproducibility debt
If you cannot reproduce a model, you cannot fix it under pressure.

Minimum: version code, data snapshots, feature definitions, training config, and model artifacts.

Monitoring debt
Teams ship with uptime monitoring and call it done.

You need:

data drift signals
prediction distribution shifts
segment-level performance when labels arrive
operational metrics (latency, errors, cost per request)
user feedback loops (complaints, overrides, appeals)
Ownership debt
If only one person understands the training pipeline, that person becomes your availability risk.

Write it down. Automate it. Rotate ownership.

The hybrid architecture I see working most often
If you want speed and control, hybrid is usually the reality.

A practical hybrid stack looks like this:

Buy a foundation model API or managed model platform
Build your retrieval layer (RAG), guardrails, and orchestration
Build your eval harness, monitoring, and audit logging
Keep sensitive data inside your boundary via redaction, retrieval controls, and least-privilege access
Use feature flags to route traffic and roll back quickly
Hybrid works when you treat the vendor as a dependency behind an interface, not as your entire system.

Where governance frameworks help developers (without slowing them down)
I am not asking engineers to become lawyers. I am asking teams to ship systems that can be defended and operated.

Three references that translate well into engineering controls:

NIST AI Risk Management Framework for lifecycle risk thinking
ISO/IEC 42001 for management system discipline (roles, controls, evidence)
EU AI Act for risk-tiered obligations where applicable
The developer translation is simple: turn requirements into pipeline gates, monitoring, and evidence artifacts.

Read the original, and then argue with me
If you want the broader operating model version, read: Building vs Buying Decisions for AI Systems

And if you want more production-focused AI engineering notes, the full blog is here: hernanhuwyler.wordpress.com

Closing question (the one I ask before approving either path)
If your AI system starts producing harmful outputs tomorrow, can you prove what changed and roll back in under 30 minutes?

The 10 Engineering Practices That Separate Production AI Systems From Science Projects

Hernan Huwyler — Mon, 13 Apr 2026 21:34:48 +0000

Managing AI development and deployment requires fundamentally different practices than traditional software engineering. AI systems derive behavior from training data distributions, not deterministic code paths. They exhibit statistical drift, emergent failure modes, and probabilistic degradation that deterministic software doesn't experience.

A model that hits 94% validation accuracy can crater to 71% in production when data distributions shift. A chatbot that passes every integration test can hallucinate confidential information in month three because training data memorization wasn't tested. A recommendation system that drives 18% revenue lift in A/B testing can amplify bias patterns that weren't visible in aggregate metrics.

Most AI projects stall because teams manage them like software projects—fixed requirements, linear development, deploy-and-forget operations. Then reality hits: training data goes stale, vendor foundation models change behavior without notice, regulators ask for explainability that wasn't architected, or users reject outputs because trust mechanisms weren't built.

Production-ready AI engineering requires practices built for experimentation under constraints, continuous distribution monitoring, automated validation pipelines, and staged deployment with statistical power analysis. This guide synthesizes technical best practices from MLOps research, regulatory frameworks, and production failure analysis into executable engineering guidance.

Learn more about managing AI development and deployment projects →

Why AI Engineering Demands Different Primitives Than Software Engineering
AI systems exhibit three properties that break traditional software engineering assumptions, requiring adapted technical practices.

First: Development is fundamentally stochastic, not deterministic. You cannot specify training convergence timelines the way you spec API endpoints. Model performance emerges from data-algorithm interactions that resist precise prediction until training completes. A technically sound architecture may fail to meet business thresholds due to insufficient training data, feature multicollinearity, or train-test distribution mismatch. Engineering workflows must accommodate this irreducible uncertainty rather than treating it as planning failure.

Second: Production behavior changes without code changes. Data drift causes model performance degradation over time even when no engineer touches the codebase. A recommendation engine behaves differently on day 500 than day 1 because user behavior evolves, seasonal patterns shift, or competitive dynamics change the action space. Deployment is the beginning of the operational lifecycle, not its end. Traditional software's deploy-and-monitor model fails for systems whose behavior is coupled to evolving external distributions.

Third: Novel failure modes demand novel testing strategies. Adversarial vulnerability, training data memorization, spurious correlation amplification, and distributional unfairness don't exist in conventional software. Testing these requires statistical validation techniques, not just unit tests and integration tests. A model can pass every software engineering quality gate while failing every ML engineering quality gate.

These three properties cascade through the entire development stack: requirements can't be fully specified upfront, timelines must include stochastic components, testing must validate statistical properties, deployment must support continuous model updates, and operations must monitor distributional shifts rather than just error rates.

Engineering primitive: Build your project management around two milestone types:

Fixed milestones: Governance approvals, security reviews, deployment dates, compliance checkpoints
Adaptive milestones: Model performance gates with go/no-go evaluation protocols
Fixed milestones maintain stakeholder accountability and cross-functional coordination. Adaptive milestones acknowledge that model development is stochastic and may require multiple training iterations to hit performance thresholds.

When you treat 0.85 F1-score as a fixed milestone with a hard deadline, teams either cut validation rigor to meet the date or blow through the timeline repeatedly. When you treat 0.85 F1-score as an adaptive gate with statistical confidence requirements and evaluation procedures, the project maintains momentum while accommodating genuine technical uncertainty.

Best Practice 1: Build Governance With Actual Decision Rights, Not Advisory Theater
Effective AI engineering starts with explicit governance structures that have real authority over three critical gates: use case approval (can we build this), deployment approval (can we ship this), and continuation approval (should we keep running this).

Define three distinct ownership roles for every AI system:

Business owner (accountable for outcomes and compliance):

Owns business case, success metrics, regulatory exposure
Bears responsibility for user impact, fairness, transparency
Authority to approve use case and define acceptable risk tradeoffs
Technical owner (responsible for model performance):

Owns architecture decisions, training methodology, validation protocols
Responsible for model accuracy, latency, resource efficiency
Authority to approve technical design and deployment readiness
Operations owner (manages production behavior):

Owns monitoring infrastructure, drift detection, incident response
Responsible for retrain triggers, rollback decisions, retirement criteria
Authority to pull systems exhibiting unacceptable degradation
These may be the same person in small teams, but the responsibilities must be explicitly assigned. Unassigned responsibilities don't get fulfilled—they become the gap where production failures hide.

Critical governance requirement: The governance structure must have authority to block deployments, not just review them. Advisory governance that can recommend against deployment while the business sponsor overrides becomes performative compliance theater.

Grant your governance structure explicit stop authority at three gates:

Use case approval: Block projects that create unacceptable regulatory risk, violate ethical constraints, or lack necessary data rights
Deployment approval: Block launches that fail validation criteria, lack adequate monitoring, or present unmitigated security vulnerabilities
Continuation approval: Mandate retirement for systems exhibiting persistent fairness failures, irremediable drift, or regulatory non-compliance
Engineering implementation:

Python

Example governance gate in CI/CD pipeline

class DeploymentGovernanceGate:
def init(self, risk_level: str):
self.risk_level = risk_level
self.required_approvals = self._get_approval_requirements()

def _get_approval_requirements(self) -> Dict[str, bool]:
    """Define required approvals based on risk classification"""
    if self.risk_level == "high":
        return {
            "technical_validation": False,
            "fairness_audit": False,
            "security_review": False,
            "legal_approval": False,
            "exec_sponsor": False
        }
    elif self.risk_level == "medium":
        return {
            "technical_validation": False,
            "fairness_audit": False,
            "security_review": False
        }
    else:  # low risk
        return {
            "technical_validation": False,
            "automated_checks": False
        }

def check_approval_status(self, approvals: Dict[str, bool]) -> Tuple[bool, List[str]]:
    """Block deployment if required approvals missing"""
    missing = [k for k, v in self.required_approvals.items() if not approvals.get(k, False)]
    can_deploy = len(missing) == 0
    return can_deploy, missing

def enforce_gate(self, approvals: Dict[str, bool]) -> None:
    """Hard block deployment without required approvals"""
    can_deploy, missing = self.check_approval_status(approvals)
    if not can_deploy:
        raise DeploymentBlockedException(
            f"Deployment blocked: missing required approvals: {missing}"
        )

This pattern enforces governance mechanically rather than relying on process compliance. The CI/CD pipeline cannot proceed without cryptographically-signed approval artifacts from required reviewers.

Best Practice 2: Implement Risk-Tiered Lifecycle Controls Based on Impact Classification
Apply governance intensity proportional to potential harm. An internal doc summarization tool doesn't need the same validation rigor as a credit decisioning model affecting millions of loan applicants.

Structure your AI lifecycle with five phases, each with documented decision gates:

Phase 1: Business case and risk classification

Define problem, expected value, success metrics before writing code
Classify regulatory risk tier (following EU AI Act categories or internal framework)
Assess data availability, representativeness, rights-to-use
Output: Approved use case with risk classification and data strategy
Phase 2: Design and data preparation

Evaluate training data quality, bias, provenance
Document data lineage, collection methodology, known limitations
Build reproducible preprocessing pipelines with version control
Output: Validated dataset with documented characteristics and preprocessing code
Phase 3: Development and validation

Train models with experiment tracking (MLflow, Weights & Biases)
Validate performance, fairness, robustness against defined criteria
Conduct adversarial testing, out-of-distribution evaluation, subgroup analysis
Output: Validated model with performance documentation and failure mode analysis
Phase 4: Deployment readiness

Verify monitoring infrastructure, alerting thresholds, rollback mechanisms
Confirm API security, rate limiting, input validation, output sanitization
Test integration with downstream systems under realistic load
Output: Production-ready system with operational runbooks and incident response procedures
Phase 5: Continuous operation

Monitor drift (data, concept, prediction), performance degradation, fairness metrics
Execute scheduled retraining or trigger-based updates with re-validation
Maintain audit logs, decision lineage, explainability artifacts
Output: Sustained production operation with documented performance history
Higher-risk systems require more intensive validation at each gate. Use a classification system to determine governance intensity:

High-risk systems (safety-critical, rights-affecting, regulated decisions):

Require independent validation by team that didn't build the model
Demand comprehensive fairness testing across demographic segments
Need documented human oversight procedures with override rates monitored
Must undergo legal, compliance, and ethics committee review
Medium-risk systems (significant business impact, indirect user effect):

Require peer review and approval from senior technical leadership
Need fairness testing for known sensitive attributes
Should have human review for edge cases and high-uncertainty predictions
Low-risk systems (internal tools, non-consequential recommendations):

Can use automated validation gates with threshold-based approval
Need basic performance testing and data quality checks
Should have monitoring but may not require dedicated operational team
Critical engineering practice: Conduct regulatory risk classification during planning, not after development. Discovering your credit model falls under FCRA requirements or your medical AI triggers FDA oversight after six months of development typically requires architectural redesign and multi-month delays.

By early 2026, over 72 countries have launched 1,000+ AI policy initiatives. The EU AI Act imposes fines up to €35M or 7% of global revenue. Map your systems against applicable regulations based on where you develop, deploy, and whose data you process.

Engineering implementation:

Python

from enum import Enum
from typing import Dict, List

class RiskTier(Enum):
PROHIBITED = "prohibited" # EU AI Act prohibited practices
HIGH = "high" # Rights-affecting, safety-critical
MEDIUM = "medium" # Significant business impact
LOW = "low" # Internal tools, minimal impact

class RegulatoryClassifier:
"""Classify AI systems against regulatory frameworks"""

def __init__(self):
    self.eu_ai_act_rules = self._load_eu_ai_act_criteria()
    self.sector_regulations = self._load_sector_regulations()

def classify_system(self, 
                   use_case: str,
                   decision_type: str,
                   affected_rights: List[str],
                   deployment_region: List[str]) -> Dict:
    """
    Classify system risk tier and applicable regulations

    Args:
        use_case: Description of AI system purpose
        decision_type: automated/human-in-loop/human-on-loop
        affected_rights: List of fundamental rights potentially impacted
        deployment_region: Geographic deployment locations

    Returns:
        Dictionary with risk tier and applicable regulations
    """
    classification = {
        "risk_tier": self._determine_risk_tier(
            use_case, decision_type, affected_rights
        ),
        "regulations": self._identify_regulations(
            use_case, deployment_region
        ),
        "required_controls": [],
        "documentation_requirements": []
    }

    # Map controls to risk tier
    classification["required_controls"] = self._get_controls_for_tier(
        classification["risk_tier"]
    )

    # Map documentation to regulations
    classification["documentation_requirements"] = self._get_docs_for_regs(
        classification["regulations"]
    )

    return classification

def _determine_risk_tier(self, use_case, decision_type, affected_rights):
    """Apply EU AI Act risk classification logic"""
    # Prohibited practices
    prohibited_patterns = [
        "social scoring",
        "subliminal manipulation",
        "exploitation of vulnerabilities"
    ]
    if any(p in use_case.lower() for p in prohibited_patterns):
        return RiskTier.PROHIBITED

    # High-risk categories
    high_risk_domains = [
        "employment",
        "education",
        "law enforcement",
        "migration",
        "justice",
        "credit scoring",
        "insurance pricing",
        "essential services"
    ]

    critical_rights = [
        "non-discrimination",
        "privacy",
        "fair trial",
        "freedom of expression"
    ]

    if (any(d in use_case.lower() for d in high_risk_domains) and
        decision_type == "automated" and
        any(r in affected_rights for r in critical_rights)):
        return RiskTier.HIGH

    # Medium/low classification logic
    if decision_type == "automated" or len(affected_rights) > 0:
        return RiskTier.MEDIUM
    return RiskTier.LOW

This systematic classification drives governance requirements, documentation standards, and validation rigor throughout the lifecycle.

Best Practice 3: Adopt MLOps as Core Engineering Infrastructure, Not Optional Tooling
MLOps isn't auxiliary tooling—it's foundational infrastructure that makes AI systems reproducible, scalable, and governable at production scale. Five MLOps components deliver measurable operational improvements.

Component 1: Data Engineering Automation
Tools: Apache Airflow, Kafka, Spark, dbt
Impact: 30% reduction in data preparation time, 25% improvement in data quality

Why it matters: Manual data pipelines don't scale and create reproducibility failures. Automated pipelines ensure consistent preprocessing, enable versioned feature engineering, and catch data quality regressions before they poison training.

Engineering pattern:

Python

Airflow DAG for reproducible data pipeline

from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.providers.amazon.aws.sensors.s3 import S3KeySensor
from datetime import datetime, timedelta
import great_expectations as ge

These tests run automatically in CI/CD. If any fairness constraint is violated or adversarial robustness is insufficient, the pipeline fails and deployment blocks.

Engineering primitive: Start MLOps adoption with version control for models, data, and configuration. This single practice addresses the reproducibility crisis that undermines AI system trust. When a production model behaves unexpectedly, version control lets you identify exactly which model artifact is running, which data it trained on, which hyperparameters produced it, and what changed between current and previous versions.

Without version control, diagnosis depends on individual memory and informal notes—which degrade rapidly as time passes and team members change. Version control is the foundation for every other MLOps practice.

Best Practice 4: Build Modular, Testable Pipelines With Automated Validation
Break AI workflows into independent, composable components: data ingestion, validation, preprocessing, feature engineering, training, evaluation, deployment, monitoring. Each component should be developable, testable, and deployable independently.

Why modularity matters:

28% faster deployment through component reuse
45% reduction in code duplication across projects
Easier debugging (isolate failures to specific components)
Team parallelization (different engineers own different components)
Engineering pattern:

Python

pipeline/components.py

from abc import ABC, abstractmethod
from dataclasses import dataclass
from typing import Any, Dict
import logging

@dataclass
class PipelineArtifact:
"""Metadata for versioned pipeline artifacts"""
data: Any
version: str
timestamp: datetime
metadata: Dict

class PipelineComponent(ABC):
"""Base class for modular pipeline components"""

def __init__(self, name: str, version: str):
    self.name = name
    self.version = version
    self.logger = logging.getLogger(f"pipeline.{name}")

@abstractmethod
def execute(self, input_artifact: PipelineArtifact) -> PipelineArtifact:
    """Execute component logic, return versioned artifact"""
    pass

def validate_input(self, artifact: PipelineArtifact) -> bool:
    """Validate input artifact meets component requirements"""
    return True  # Override in subclasses

def log_execution(self, input_artifact, output_artifact):
    """Log component execution for lineage tracking"""
    mlflow.log_params({
        f"{self.name}_input_version": input_artifact.version,
        f"{self.name}_output_version": output_artifact.version,
        f"{self.name}_component_version": self.version
    })

class DataIngestion(PipelineComponent):
"""Fetch raw data from source systems"""

def __init__(self, source_config: Dict):
    super().__init__(name="data_ingestion", version="1.2.0")
    self.source_config = source_config

def execute(self, input_artifact: PipelineArtifact) -> PipelineArtifact:
    self.logger.info(f"Ingesting data from {self.source_config['source']}")

    # Fetch data
    raw_data = self._fetch_from_source()

    # Create versioned artifact
    artifact = PipelineArtifact(
        data=raw_data,
        version=f"raw_{datetime.now().strftime('%Y%m%d_%H%M%S')}",
        timestamp=datetime.now(),
        metadata={
            "source": self.source_config['source'],
            "row_count": len(raw_data),
            "component_version": self.version
        }
    )

    self.log_execution(input_artifact, artifact)
    return artifact

class DataValidation(PipelineComponent):
"""Validate data quality using Great Expectations"""

def __init__(self, expectation_suite: str):
    super().__init__(name="data_validation", version="1.1.0")
    self.expectation_suite = expectation_suite

def execute(self, input_artifact: PipelineArtifact) -> PipelineArtifact:
    self.logger.info("Validating data quality")

    # Run Great Expectations validation
    validation_results = self._run_expectations(input_artifact.data)

    if not validation_results["success"]:
        failed_expectations = validation_results["failed_expectations"]
        raise DataQualityException(
            f"Data validation failed: {failed_expectations}"
        )

    # Pass through data with validation metadata
    artifact = PipelineArtifact(
        data=input_artifact.data,
        version=f"{input_artifact.version}_validated",
        timestamp=datetime.now(),
        metadata={
            **input_artifact.metadata,
            "validation_suite": self.expectation_suite,
            "validation_passed": True,
            "validation_timestamp": datetime.now().isoformat()
        }
    )

    self.log_execution(input_artifact, artifact)
    return artifact

class FeatureEngineering(PipelineComponent):
"""Transform raw data into model features"""

def __init__(self, transform_config: Dict):
    super().__init__(name="feature_engineering", version="2.3.1")
    self.transform_config = transform_config

def execute(self, input_artifact: PipelineArtifact) -> PipelineArtifact:
    self.logger.info("Engineering features")

    # Apply transformations
    features = self._apply_transforms(input_artifact.data)

    # Store feature statistics for drift detection
    feature_stats = self._compute_statistics(features)

    artifact = PipelineArtifact(
        data=features,
        version=f"features_v{self.version}_{datetime.now().strftime('%Y%m%d')}",
        timestamp=datetime.now(),
        metadata={
            "input_version": input_artifact.version,
            "transform_config": self.transform_config,
            "feature_count": features.shape[1],
            "feature_statistics": feature_stats,
            "component_version": self.version
        }
    )

    self.log_execution(input_artifact, artifact)
    return artifact

Pipeline orchestration

class Pipeline:
"""Orchestrate modular components into complete workflow"""

def __init__(self, components: List[PipelineComponent]):
    self.components = components

def execute(self, initial_input: PipelineArtifact = None) -> PipelineArtifact:
    """Run all components in sequence"""
    artifact = initial_input or PipelineArtifact(
        data=None, version="initial", timestamp=datetime.now(), metadata={}
    )

    for component in self.components:
        try:
            artifact = component.execute(artifact)
        except Exception as e:
            logging.error(
                f"Pipeline failed at component {component.name}: {e}"
            )
            raise

    return artifact

Usage

training_pipeline = Pipeline(components=[
DataIngestion(source_config={"source": "s3://training-data"}),
DataValidation(expectation_suite="training_data_expectations"),
FeatureEngineering(transform_config={"version": "2.3.1"}),
ModelTraining(hyperparameters={"n_estimators": 200}),
ModelValidation(validation_suite="model_performance_tests"),
])

final_artifact = training_pipeline.execute()
Each component is independently testable, reusable across projects, and generates lineage metadata automatically.

What to automate in testing:

Data integrity tests: Schema validation, range checks, null rate limits, distribution similarity
Model performance tests: Accuracy/F1/precision/recall against thresholds on holdout data
Fairness tests: Demographic parity, equalized odds across protected attributes
Integration tests: Model outputs flow correctly to downstream systems
Robustness tests: Adversarial examples, out-of-distribution inputs, edge cases
Engineering primitive: The highest-ROI testing practice is automated data validation at pipeline ingestion. Most production AI failures originate from data problems (unexpected nulls, format changes, distribution shifts, corrupted feeds), not model problems.

Build validation rules for every input field: acceptable ranges, expected data types, maximum null rates, distribution similarity to training data. When any rule is violated, pipeline pauses and alerts data engineering. This single control prevents cascading failures where bad data → bad predictions → bad business decisions before anyone notices data degradation.

Learn more about comprehensive AI project management practices →

Best Practice 5: Manage Third-Party AI With Same Rigor as Internal Models
Most organizations acquire more AI than they build. AI is embedded in vendor SaaS (Salesforce Einstein, HubSpot predictions, SAP intelligent automation), procurement platforms, HR systems, and enterprise software. Each embedded AI component carries risks the organization remains accountable for regardless of who built it.

Third-party AI governance requires four technical disciplines:

Pre-Procurement Technical Due Diligence Before signing contracts, evaluate:

Model development practices:

Training methodology documented?
Validation approach adequate for use case?
Bias testing conducted across demographic segments?
Performance metrics reported with confidence intervals?
Training data provenance:

Data sources disclosed?
Data collection methodology ethical and legal?
Known representativeness gaps documented?
Data refresh/update cadence defined?
Security and robustness:

Adversarial testing conducted?
Input validation implemented?
Rate limiting and abuse prevention?
Incident response procedures documented?
Technical implementation:

Python

vendor_evaluation_framework.py

from dataclasses import dataclass
from typing import List, Dict
from enum import Enum

class RiskLevel(Enum):
LOW = "low"
MEDIUM = "medium"
HIGH = "high"
CRITICAL = "critical"

@dataclass
class VendorAIEvaluation:
"""Framework for assessing vendor AI components"""

vendor_name: str
ai_component: str
use_case: str

# Technical assessment
model_documentation_quality: RiskLevel
training_data_transparency: RiskLevel
performance_validation_rigor: RiskLevel
bias_testing_adequacy: RiskLevel
security_robustness: RiskLevel

# Operational assessment
monitoring_capabilities: RiskLevel
update_notification_process: RiskLevel
incident_response_maturity: RiskLevel
data_portability: RiskLevel

# Legal assessment
liability_allocation: RiskLevel
compliance_coverage: RiskLevel
audit_rights: RiskLevel

def overall_risk_score(self) -> float:
    """Calculate weighted risk score"""
    weights = {
        "model_documentation_quality": 0.10,
        "training_data_transparency": 0.10,
        "performance_validation_rigor": 0.15,
        "bias_testing_adequacy": 0.15,
        "security_robustness": 0.10,
        "monitoring_capabilities": 0.10,
        "update_notification_process": 0.05,
        "incident_response_maturity": 0.10,
        "data_portability": 0.05,
        "liability_allocation": 0.05,
        "compliance_coverage": 0.03,
        "audit_rights": 0.02
    }

    risk_values = {
        RiskLevel.LOW: 1,
        RiskLevel.MEDIUM: 2,
        RiskLevel.HIGH: 3,
        RiskLevel.CRITICAL: 4
    }

    score = 0
    for field, weight in weights.items():
        risk_level = getattr(self, field)
        score += weight * risk_values[risk_level]

    return score

def approval_recommendation(self) -> str:
    """Recommend procurement decision"""
    score = self.overall_risk_score()

    if score < 1.5:
        return "APPROVED"
    elif score < 2.5:
        return "APPROVED_WITH_CONDITIONS"
    elif score < 3.0:
        return "REQUIRES_REMEDIATION"
    else:
        return "REJECTED"

Contractual Provisions for Transparency and Control Negotiate contracts that include:

Performance guarantees:

Minimum accuracy/precision/recall thresholds
Maximum latency commitments (P95, P99)
Uptime SLAs
Financial penalties for persistent underperformance
Change notification requirements:

30-60 day notice before model updates
Disclosure of material algorithm changes
Performance impact assessment for updates
Right to defer updates that degrade performance
Audit and transparency rights:

Annual model card updates
Access to performance metrics on customer's data
Right to conduct independent validation
Explanation of prediction rationale for high-stakes decisions
Data and exit rights:

Data ownership clearly allocated
Data portability in machine-readable formats
Model export or API access post-contract
Reasonable transition assistance period
Example contract language:

text

VENDOR AI TRANSPARENCY AND GOVERNANCE ADDENDUM

Model Documentation
Vendor shall provide and maintain current:
- Model card documenting intended use, known limitations, performance metrics
- Description of training data sources, collection methodology, known biases
- Validation methodology and results on representative test datasets
- Update frequency: Annually minimum, within 30 days of material changes
Performance Commitments
Vendor commits to minimum performance thresholds measured on Customer's data:
- Accuracy: 85% (±2%)
- Latency P95: 200ms
- Latency P99: 500ms
- Uptime: 99.5%

Performance measured quarterly. Persistent underperformance (2 consecutive quarters
below threshold) triggers service credits of [X]% monthly fees per threshold violation.

Change Management
- Material algorithm changes require 60-day advance notice
- Notice must include expected performance impact assessment
- Customer may defer updates up to 90 days for internal testing
- Emergency security updates may proceed with 48-hour notice
Fairness and Bias
- Vendor shall conduct annual bias testing across [specified demographic attributes]
- Results reported to Customer within 30 days of completion
- Bias exceeding [X]% demographic parity triggers remediation plan
- Customer may conduct independent fairness audits annually
Data Rights and Exit
- Customer retains all rights to input data and derived analytics
- Upon termination, Vendor provides:
  - Complete data export in CSV/JSON within 30 days
  - API access continuation for 90-day transition period
  - Documentation of any Customer-specific model tuning
- Vendor deletes all Customer data within 60 days of termination
Independent Monitoring of Vendor AI Performance
Don't rely solely on vendor-reported metrics. Build independent monitoring that tracks vendor AI performance on your data and your use case.

Engineering pattern:

Python

vendor_ai_monitor.py

import pandas as pd
import numpy as np
from typing import Dict, List
from dataclasses import dataclass
from datetime import datetime, timedelta

@dataclass
class VendorPerformanceBaseline:
"""Expected performance based on contract/validation"""
accuracy: float
precision: float
recall: float
latency_p95_ms: float
latency_p99_ms: float

class VendorAIMonitor:
"""Monitor third-party AI component performance"""

def __init__(self, vendor_name: str, component_name: str, 
             baseline: VendorPerformanceBaseline):
    self.vendor_name = vendor_name
    self.component_name = component_name
    self.baseline = baseline
    self.performance_history = []

def log_prediction(self, 
                   prediction: Any,
                   ground_truth: Any = None,
                   latency_ms: float = None,
                   timestamp: datetime = None):
    """Log individual predictions for aggregate analysis"""
    self.performance_history.append({
        "timestamp": timestamp or datetime.now(),
        "prediction": prediction,
        "ground_truth": ground_truth,
        "latency_ms": latency_ms
    })

def compute_weekly_performance(self) -> Dict:
    """Aggregate performance over rolling week"""
    df = pd.DataFrame(self.performance_history)
    week_ago = datetime.now() - timedelta(days=7)
    recent = df[df['timestamp'] > week_ago]

    # Filter to records with ground truth
    labeled = recent[recent['ground_truth'].notna()]

    if len(labeled) < 100:
        return {"status": "insufficient_data", "sample_size": len(labeled)}

    # Compute performance metrics
    from sklearn.metrics import accuracy_score, precision_score, recall_score

    performance = {
        "accuracy": accuracy_score(labeled['ground_truth'], labeled['prediction']),
        "precision": precision_score(labeled['ground_truth'], labeled['prediction']),
        "recall": recall_score(labeled['ground_truth'], labeled['prediction']),
        "latency_p95_ms": recent['latency_ms'].quantile(0.95),
        "latency_p99_ms": recent['latency_ms'].quantile(0.99),
        "sample_size": len(labeled),
        "timestamp": datetime.now()
    }

    return performance

def detect_sla_violations(self, current_performance: Dict) -> List[str]:
    """Check performance against contracted SLAs"""
    violations = []
    tolerance = 0.02  # 2% tolerance for statistical noise

    if current_performance["accuracy"] < self.baseline.accuracy - tolerance:
        violations.append(
            f"Accuracy SLA violation: {current_performance['accuracy']:.3f} "
            f"< {self.baseline.accuracy:.3f}"
        )

    if current_performance["latency_p95_ms"] > self.baseline.latency_p95_ms * 1.2:
        violations.append(
            f"Latency P95 SLA violation: {current_performance['latency_p95_ms']:.1f}ms "
            f"> {self.baseline.latency_p95_ms:.1f}ms"
        )

    return violations

def generate_vendor_performance_report(self) -> str:
    """Generate report for vendor accountability discussions"""
    current = self.compute_weekly_performance()
    violations = self.detect_sla_violations(current)

    report = f"""
    Vendor AI Performance Report
    ============================
    Vendor: {self.vendor_name}
    Component: {self.component_name}
    Period: Past 7 days
    Sample Size: {current['sample_size']}

    Performance vs. Baseline:
    - Accuracy: {current['accuracy']:.3f} (baseline: {self.baseline.accuracy:.3f})
    - Precision: {current['precision']:.3f} (baseline: {self.baseline.precision:.3f})
    - Recall: {current['recall']:.3f} (baseline: {self.baseline.recall:.3f})
    - Latency P95: {current['latency_p95_ms']:.1f}ms (baseline: {self.baseline.latency_p95_ms:.1f}ms)

    SLA Status: {"VIOLATED" if violations else "COMPLIANT"}
    """

    if violations:
        report += "\nViolations:\n" + "\n".join(f"- {v}" for v in violations)

    return report

Shadow AI Detection and Approved Alternative Provision When employees adopt AI tools outside formal channels (personal ChatGPT for work tasks, unauthorized browser extensions, AI plugins), they create unmanaged risk. Detection plus approved alternatives works better than prohibition.

Detection mechanisms:

Network monitoring for API calls to known AI services
Browser extension inventory tools
Data loss prevention (DLP) alerts for sensitive data sent to external AI
User surveys asking what tools they actually use
Approved alternatives:

Enterprise ChatGPT with data residency guarantees
Copilot Business with admin controls
Internal model deployments for common use cases
Self-service AI catalog with pre-approved, governed tools
Engineering primitive: Build a third-party AI inventory cataloging every vendor component operating in your environment, including AI embedded in SaaS platforms not marketed as "AI products."

Most organizations discover during first inventory that they have 3-5× more third-party AI than they knew about, because vendors added AI features through routine software updates without prominent disclosure.

Action: Review release notes from your top 20 software vendors for past 18 months. Many added AI features (smart recommendations, automated classification, predictive analytics, chatbots) without labeling them as "AI." Each is a third-party AI component requiring governance.

Best Practice 6: Deploy in Phases With Statistical Validation at Each Stage
Rush from prototype to full production and you deploy untested assumptions at scale. Phased deployment with statistical validation catches problems when they're cheap to fix.

Three-phase deployment pattern:

Phase 1: Shadow Mode (2-4 weeks)
Model runs in production environment but outputs aren't used for decisions. Compare AI predictions to current process/human decisions.

Purpose:

Validate production data pipeline works
Measure actual latency under real load
Identify data quality issues missed in development
Establish performance baseline on production distribution
Success criteria:

Pipeline processes 100% of production volume without failures
Latency P95 < threshold
Performance metrics within 5% of validation results
No critical data quality alerts
Engineering implementation:

Python

shadow_deployment.py

class ShadowDeployment:
"""Run model in shadow mode for validation"""

def __init__(self, model, baseline_system, metrics_logger):
    self.model = model
    self.baseline = baseline_system
    self.metrics = metrics_logger

def process_request(self, input_data: Dict) -> Dict:
    """Process request through both shadow model and baseline"""

    # Get baseline decision (current production system)
    baseline_start = time.time()
    baseline_decision = self.baseline.predict(input_data)
    baseline_latency = (time.time() - baseline_start) * 1000

    # Get shadow model prediction (not used for actual decision)
    shadow_start = time.time()
    shadow_prediction = self.model.predict(input_data)
    shadow_latency = (time.time() - shadow_start) * 1000

    # Log for comparison analysis
    self.metrics.log({
        "timestamp": datetime.now(),
        "baseline_decision": baseline_decision,
        "shadow_prediction": shadow_prediction,
        "baseline_latency_ms": baseline_latency,
        "shadow_latency_ms": shadow_latency,
        "agreement": baseline_decision == shadow_prediction
    })

    # Return baseline decision (shadow doesn't affect production)
    return {"decision": baseline_decision, "mode": "baseline"}

def generate_shadow_analysis(self, days: int = 7) -> Dict:
    """Analyze shadow mode performance"""
    logs = self.metrics.get_logs(days=days)

    return {
        "total_requests": len(logs),
        "shadow_latency_p95": np.percentile(logs['shadow_latency_ms'], 95),
        "shadow_latency_p99": np.percentile(logs['shadow_latency_ms'], 99),
        "baseline_latency_p95": np.percentile(logs['baseline_latency_ms'], 95),
        "agreement_rate": logs['agreement'].mean(),
        "shadow_error_rate": logs['shadow_error'].mean() if 'shadow_error' in logs else 0,
    }

Phase 2: Canary Deployment (1-2 weeks)
Route small percentage of production traffic (5-10%) to new model. Monitor performance, errors, user feedback. Statistically compare canary to baseline.

Purpose:

Detect unexpected behaviors at limited scale
Measure business impact on real users
Validate monitoring and rollback mechanisms work
Build confidence before full rollout
Success criteria:

Performance on canary traffic matches shadow mode performance
Error rate < baseline error rate + tolerance
No critical user complaints
Business metrics (conversion, revenue, satisfaction) neutral or positive
Engineering implementation:

Python

canary_deployment.py

from scipy import stats

class CanaryDeployment:
"""Gradual rollout with statistical validation"""

def __init__(self, baseline_model, canary_model, 
             canary_percentage: float = 0.05):
    self.baseline = baseline_model
    self.canary = canary_model
    self.canary_pct = canary_percentage
    self.metrics = {
        "baseline": {"predictions": [], "errors": [], "latencies": []},
        "canary": {"predictions": [], "errors": [], "latencies": []}
    }

def route_request(self, user_id: str) -> str:
    """Deterministically route user to baseline or canary"""
    # Use consistent hashing so same user always sees same model
    import hashlib
    hash_val = int(hashlib.md5(user_id.encode()).hexdigest(), 16)
    return "canary" if (hash_val % 100) < (self.canary_pct * 100) else "baseline"

def process_request(self, user_id: str, input_data: Dict) -> Dict:
    """Route request and track metrics"""
    variant = self.route_request(user_id)
    model = self.canary if variant == "canary" else self.baseline

    start = time.time()
    try:
        prediction = model.predict(input_data)
        error = False
    except Exception as e:
        logging.error(f"Model error in {variant}: {e}")
        prediction = None
        error = True

    latency = (time.time() - start) * 1000

    self.metrics[variant]["predictions"].append(prediction)
    self.metrics[variant]["errors"].append(error)
    self.metrics[variant]["latencies"].append(latency)

    return {"prediction": prediction, "variant": variant}

def statistical_comparison(self) -> Dict:
    """Compare canary to baseline with statistical tests"""
    baseline_errors = self.metrics["baseline"]["errors"]
    canary_errors = self.metrics["canary"]["errors"]

    # Error rate comparison (binomial test)
    baseline_error_rate = np.mean(baseline_errors)
    canary_error_rate = np.mean(canary_errors)

    # Two-proportion z-test
    n1, n2 = len(baseline_errors), len(canary_errors)
    p1, p2 = baseline_error_rate, canary_error_rate
    p_pooled = (n1*p1 + n2*p2) / (n1 + n2)
    se = np.sqrt(p_pooled * (1-p_pooled) * (1/n1 + 1/n2))
    z_score = (p2 - p1) / se if se > 0 else 0
    p_value = 2 * (1 - stats.norm.cdf(abs(z_score)))

    # Latency comparison (Mann-Whitney U test)
    baseline_latencies = self.metrics["baseline"]["latencies"]
    canary_latencies = self.metrics["canary"]["latencies"]
    latency_stat, latency_p = stats.mannwhitneyu(
        baseline_latencies, canary_latencies, alternative='two-sided'
    )

    return {
        "baseline_error_rate": baseline_error_rate,
        "canary_error_rate": canary_error_rate,
        "error_rate_difference": canary_error_rate - baseline_error_rate,
        "error_rate_p_value": p_value,
        "error_rate_significant": p_value < 0.05,
        "baseline_latency_p50": np.median(baseline_latencies),
        "canary_latency_p50": np.median(canary_latencies),
        "latency_p_value": latency_p,
        "latency_significant": latency_p < 0.05,
        "recommendation": self._get_recommendation(
            canary_error_rate, baseline_error_rate, p_value
        )
    }

def _get_recommendation(self, canary_err, baseline_err, p_value):
    """Recommend continue/rollback based on statistical evidence"""
    MAX_ACCEPTABLE_ERROR_INCREASE = 0.005  # 0.5 percentage points

    if canary_err > baseline_err + MAX_ACCEPTABLE_ERROR_INCREASE:
        if p_value < 0.05:
            return "ROLLBACK_IMMEDIATELY"
        else:
            return "MONITOR_CLOSELY"
    else:
        return "PROCEED_TO_FULL_ROLLOUT"

Phase 3: Full Production (gradual traffic increase)
Gradually increase traffic to new model: 5% → 25% → 50% → 100% over days or weeks, with statistical validation at each step.

Success criteria:

Performance remains stable as traffic increases
Business metrics show improvement or neutrality
No increase in user complaints or support tickets
Monitoring dashboards show expected behavior
Rollback triggers:

Error rate increase > 0.5 percentage points (statistically significant)
Latency P95 increase > 50ms
Business metric degradation > 5%
Critical fairness violation detected
Security incident related to model
Engineering primitive: Define success criteria and rollback triggers before deployment, not during incidents. Write these as executable code with automatic rollback, not as judgment calls made under pressure.

Python

automatic_rollback.py

class AutomaticRollback:
"""Automated rollback based on monitoring thresholds"""

def __init__(self, deployment, thresholds: Dict):
    self.deployment = deployment
    self.thresholds = thresholds
    self.check_interval_seconds = 300  # 5 minutes

def monitor_and_rollback_if_needed(self):
    """Continuous monitoring with automatic rollback"""
    while True:
        time.sleep(self.check_interval_seconds)

        metrics = self.deployment.get_current_metrics()
        violations = self._check_thresholds(metrics)

        if violations:
            logging.critical(f"Threshold violations detected: {violations}")
            self._execute_rollback()
            self._alert_oncall_team(violations)
            break

def _check_thresholds(self, metrics: Dict) -> List[str]:
    """Check metrics against rollback thresholds"""
    violations = []

    if metrics["error_rate"] > self.thresholds["max_error_rate"]:
        violations.append(
            f"Error rate {metrics['error_rate']:.4f} > "
            f"threshold {self.thresholds['max_error_rate']:.4f}"
        )

    if metrics["latency_p95_ms"] > self.thresholds["max_latency_p95_ms"]:
        violations.append(
            f"Latency P95 {metrics['latency_p95_ms']:.1f}ms > "
            f"threshold {self.thresholds['max_latency_p95_ms']:.1f}ms"
        )

    return violations

def _execute_rollback(self):
    """Rollback to previous model version"""
    logging.info("Executing automatic rollback")
    self.deployment.rollback_to_previous_version()
    logging.info("Rollback completed successfully")

Learn more about comprehensive deployment strategies →

Best Practice 7: Integrate Human Oversight With Measurable Effectiveness
Human-in-the-loop processes sound good in governance documents but often fail in practice due to automation bias, time pressure, or inadequate training. Build human oversight that actually functions.

Design patterns for effective oversight:

Pattern 1: Independent review before AI recommendation
Present case facts to human reviewer first, collect their independent judgment, then show AI recommendation. Prevents automation bias where reviewers defer to AI even when their own assessment differs.

Python

human_in_loop.py

class IndependentHumanReview:
"""Collect human judgment before showing AI output"""

def review_case(self, case_data: Dict, model) -> Dict:
    """Two-stage review process"""

    # Stage 1: Human reviews case without AI
    human_review_ui = self.display_case(case_data)
    human_decision = self.collect_human_judgment(human_review_ui)
    human_confidence = self.collect_confidence_rating(human_review_ui)

    # Stage 2: Show AI recommendation
    ai_prediction = model.predict(case_data)
    ai_confidence = model.predict_proba(case_data).max()

    # Stage 3: Final decision with disagreement flag
    final_decision_ui = self.display_both_judgments(
        human_decision, human_confidence,
        ai_prediction, ai_confidence
    )
    final_decision = self.collect_final_decision(final_decision_ui)

    # Log for analysis
    return {
        "case_id": case_data["id"],
        "human_initial_decision": human_decision,
        "human_confidence": human_confidence,
        "ai_prediction": ai_prediction,
        "ai_confidence": ai_confidence,
        "final_decision": final_decision,
        "human_changed_mind": human_decision != final_decision,
        "disagreement": human_decision != ai_prediction,
        "timestamp": datetime.now()
    }

Pattern 2: Mandatory review for high-uncertainty cases
Route cases where model confidence is low to human review automatically.

Python

CONFIDENCE_THRESHOLD = 0.75

def should_require_human_review(prediction_proba: np.ndarray) -> bool:
"""Require review when model is uncertain"""
max_confidence = prediction_proba.max()
return max_confidence < CONFIDENCE_THRESHOLD

Usage in prediction pipeline

def make_decision(input_data: Dict, model) -> Dict:
prediction_proba = model.predict_proba(input_data)
prediction = model.classes_[prediction_proba.argmax()]

if should_require_human_review(prediction_proba):
    # Route to human review queue
    result = route_to_human_review(input_data, prediction, prediction_proba)
    return {"decision": result, "mode": "human_review"}
else:
    # Automated decision
    return {"decision": prediction, "mode": "automated"}

Pattern 3: Sample-based audit of automated decisions
Even when automating high-confidence predictions, randomly sample X% for post-hoc human audit.

Python

AUDIT_SAMPLE_RATE = 0.05 # 5% random sample

def make_decision_with_audit_sampling(input_data, model):
prediction = model.predict(input_data)

# Make decision
decision = {"prediction": prediction, "mode": "automated", "timestamp": datetime.now()}

# Random sampling for audit
if random.random() < AUDIT_SAMPLE_RATE:
    queue_for_audit(input_data, prediction)
    decision["queued_for_audit"] = True

return decision

Measure override rates to detect passive compliance:

If human reviewers override AI recommendations < 2-3%, investigate whether oversight is genuine (AI is consistently correct) or passive (reviewers rubber-stamp without evaluating).

Python

oversight_effectiveness_monitor.py

class OversightEffectivenessMonitor:
"""Monitor whether human oversight is functioning or performative"""

def analyze_override_patterns(self, review_logs: pd.DataFrame) -> Dict:
    """Detect passive oversight patterns"""

    # Overall override rate
    override_rate = (review_logs['human_decision'] != 
                    review_logs['ai_prediction']).mean()

    # Override rate by reviewer
    by_reviewer = review_logs.groupby('reviewer_id').apply(
        lambda x: (x['human_decision'] != x['ai_prediction']).mean()
    )

    # Override rate by time of day (fatigue indicator)
    review_logs['hour'] = review_logs['timestamp'].dt.hour
    by_hour = review_logs.groupby('hour').apply(
        lambda x: (x['human_decision'] != x['ai_prediction']).mean()
    )

    # Override rate by workload (volume indicator)
    review_logs['daily_volume'] = review_logs.groupby(
        review_logs['timestamp'].dt.date
    )['case_id'].transform('count')

    high_volume_days = review_logs[review_logs['daily_volume'] > 
                                   review_logs['daily_volume'].quantile(0.75)]
    low_volume_days = review_logs[review_logs['daily_volume'] < 
                                  review_logs['daily_volume'].quantile(0.25)]

    high_volume_override = (high_volume_days['human_decision'] != 
                           high_volume_days['ai_prediction']).mean()
    low_volume_override = (low_volume_days['human_decision'] != 
                          low_volume_days['ai_prediction']).mean()

    # Diagnose passive oversight patterns
    warnings = []

    if override_rate < 0.02:
        warnings.append(
            f"Very low override rate ({override_rate:.1%}) suggests possible "
            "automation bias or insufficient reviewer training"
        )

    if (by_reviewer < 0.01).sum() > len(by_reviewer) * 0.3:
        warnings.append(
            f"{(by_reviewer < 0.01).sum()} reviewers have <1% override rate, "
            "indicating potential rubber-stamping"
        )

    if high_volume_override < low_volume_override * 0.5:
        warnings.append(
            f"Override rate drops {(1 - high_volume_override/low_volume_override):.1%} "
            "on high-volume days, indicating workload pressure affects quality"
        )

    return {
        "overall_override_rate": override_rate,
        "override_by_reviewer": by_reviewer.to_dict(),
        "override_by_hour": by_hour.to_dict(),
        "high_volume_override_rate": high_volume_override,
        "low_volume_override_rate": low_volume_override,
        "warnings": warnings
    }

Engineering primitive: Analyze override patterns (who overrides, when, under what conditions) to distinguish active oversight from passive compliance. Override rates < 2% combined with no variation by reviewer or workload indicate performative oversight that won't catch problems.

Best Practice 8: Monitor Drift Continuously With Automated Response Workflows
Models degrade as distributions shift. Without automated drift detection and response, you discover degradation through user complaints or business impact rather than proactive alerts.

Four drift types to monitor:

Data Drift (Input Distribution Shifts) Statistical properties of production inputs diverge from training data. Model receives inputs it wasn't trained to handle well.

Detection: Kolmogorov-Smirnov test for continuous features, Chi-squared test for categorical features, Population Stability Index (PSI).

Python

drift_detection.py

from scipy.stats import ks_2samp, chi2_contingency
import numpy as np

def detect_continuous_feature_drift(training_data: np.ndarray,
production_data: np.ndarray,
significance_level: float = 0.05) -> Dict:
"""Detect drift in continuous features using KS test"""
ks_stat, p_value = ks_2samp(training_data, production_data)

is_drifted = p_value < significance_level

return {
    "ks_statistic": ks_stat,
    "p_value": p_value,
    "is_drifted": is_drifted,
    "drift_severity": "high" if ks_stat > 0.2 else ("medium" if ks_stat > 0.1 else "low")
}

def compute_psi(training_data: np.ndarray,
production_data: np.ndarray,
buckets: int = 10) -> float:
"""
Compute Population Stability Index

PSI < 0.1: No significant change
0.1 <= PSI < 0.2: Moderate change, investigate
PSI >= 0.2: Significant change, likely requires retraining
"""
# Create buckets based on training data distribution
breakpoints = np.linspace(
    training_data.min(), training_data.max(), buckets + 1
)

# Compute distributions
train_dist, _ = np.histogram(training_data, bins=breakpoints)
prod_dist, _ = np.histogram(production_data, bins=breakpoints)

# Normalize to probabilities
train_pct = train_dist / len(training_data)
prod_pct = prod_dist / len(production_data)

# Avoid division by zero
train_pct = np.where(train_pct == 0, 0.0001, train_pct)
prod_pct = np.where(prod_pct == 0, 0.0001, prod_pct)

# PSI formula: sum((prod% - train%) * ln(prod% / train%))
psi = np.sum((prod_pct - train_pct) * np.log(prod_pct / train_pct))

return psi

Concept Drift (Input-Output Relationship Changes) Relationship between features and target shifts. What predicted outcome Y given features X in training no longer holds.

Detection: Performance degradation on recent labeled data, comparison of prediction distributions over time.

Python

def detect_concept_drift(historical_performance: List[float],
current_performance: float,
window_size: int = 4,
threshold: float = 0.05) -> bool:
"""
Detect concept drift through performance degradation

Args:
    historical_performance: List of recent performance metrics
    current_performance: Latest performance measurement
    window_size: Number of periods to compare
    threshold: Acceptable performance drop

Returns:
    True if concept drift detected
"""
if len(historical_performance) < window_size:
    return False

recent_avg = np.mean(historical_performance[-window_size:])
degradation = recent_avg - current_performance

return degradation > threshold

Prediction Drift (Output Distribution Shifts) Model's prediction distribution changes even without input changes. Can indicate model instability or training issues.

Python

def detect_prediction_drift(baseline_predictions: np.ndarray,
current_predictions: np.ndarray) -> Dict:
"""Monitor distribution of model outputs"""

# For classification: compare class distribution
baseline_dist = np.bincount(baseline_predictions) / len(baseline_predictions)
current_dist = np.bincount(current_predictions) / len(current_predictions)

# JS divergence (symmetric KL divergence)
m = (baseline_dist + current_dist) / 2
js_div = 0.5 * (
    np.sum(baseline_dist * np.log(baseline_dist / m)) +
    np.sum(current_dist * np.log(current_dist / m))
)

return {
    "js_divergence": js_div,
    "is_drifted": js_div > 0.1,  # threshold
    "baseline_distribution": baseline_dist.tolist(),
    "current_distribution": current_dist.tolist()
}

Automated Response Workflows Don't just detect drift—define automated responses.

Python

drift_response.py

class DriftResponseWorkflow:
"""Automated responses to detected drift"""

def __init__(self, model_name: str, alert_config: Dict):
    self.model_name = model_name
    self.alert_config = alert_config

def handle_drift_event(self, drift_report: Dict):
    """Execute response based on drift severity"""
    severity = self._assess_severity(drift_report)

    if severity == "critical":
        self._critical_drift_response(drift_report)
    elif severity == "high":
        self._high_drift_response(drift_report)
    elif severity == "medium":
        self._medium_drift_response(drift_report)
    else:
        self._low_drift_response(drift_report)

def _assess_severity(self, drift_report: Dict) -> str:
    """Classify drift severity"""
    psi = drift_report.get("psi", 0)
    perf_degradation = drift_report.get("performance_degradation", 0)

    if psi > 0.3 or perf_degradation > 0.10:
        return "critical"
    elif psi > 0.2 or perf_degradation > 0.05:
        return "high"
    elif psi > 0.1 or perf_degradation > 0.03:
        return "medium"
    else:
        return "low"

def _critical_drift_response(self, drift_report):
    """Immediate action for critical drift"""
    # 1. Alert on-call team immediately
    self.send_alert(
        severity="critical",
        message=f"Critical drift detected in {self.model_name}",
        details=drift_report
    )

    # 2. Auto-escalate to human review
    self.enable_human_review_mode()

    # 3. Trigger emergency retraining
    self.queue_retraining_job(priority="urgent")

    # 4. Consider automatic rollback
    if drift_report["performance_degradation"] > 0.15:
        self.execute_rollback()

def _high_drift_response(self, drift_report):
    """Escalated response for high drift"""
    self.send_alert(severity="high", message=f"High drift in {self.model_name}")
    self.queue_retraining_job(priority="high")
    self.increase_monitoring_frequency()

def _medium_drift_response(self, drift_report):
    """Standard response for medium drift"""
    self.send_alert(severity="medium", message=f"Medium drift in {self.model_name}")
    self.queue_retraining_job(priority="normal")

def _low_drift_response(self, drift_report):
    """Monitoring-only response for low drift"""
    self.log_drift_event(drift_report)

Engineering primitive: Build monitoring to detect trends, not just threshold breaches. A model dropping 0.3% accuracy daily doesn't breach a 5% threshold for 16 days. Trend detection flagging sustained directional movement over 5-7 days catches gradual degradation in one-third the time.

Python

def detect_performance_trend(performance_history: pd.Series,
window_days: int = 7,
significance: float = 0.05) -> Dict:
"""Detect downward performance trends before threshold breach"""

if len(performance_history) < window_days:
    return {"trend_detected": False}

recent = performance_history.tail(window_days)

# Linear regression on recent performance
from scipy import stats
x = np.arange(len(recent))
slope, intercept, r_value, p_value, std_err = stats.linregress(x, recent.values)

# Negative slope with statistical significance indicates downward trend
is_declining = slope < 0 and p_value < significance

# Project where performance will be in 7 days if trend continues
projected_performance = intercept + slope * (len(recent) + 7)

return {
    "trend_detected": is_declining,
    "slope": slope,
    "p_value": p_value,
    "current_performance": recent.iloc[-1],
    "projected_7d_performance": projected_performance,
    "recommendation": "RETRAIN_SOON" if is_declining else "CONTINUE_MONITORING"
}

Best Practice 9: Build AI Literacy Through Cross-Functional Collaboration
Effective AI governance requires shared understanding across roles. Technical teams alone can't govern because they lack business and regulatory context. Business teams alone can't govern because they lack technical understanding. The solution is cross-functional literacy, not separate training silos.

Most effective literacy investment: Cross-functional workshop sessions where technical and business teams work through real scenarios together.

Workshop format:

Session structure (2 hours):

Technical team presents model card for real production system (15 min)
Compliance team presents regulatory requirements for same system (15 min)
Cross-functional discussion of alignment/gaps (30 min)
Hypothetical incident scenario walkthrough (45 min)
Lessons learned and action items (15 min)
Example incident scenario:

text

Scenario: Credit Decisioning Model Fairness Incident

Background:

Model approves/denies small business loan applications
Deployed 6 months ago, processing 500 applications/day
Model card documents 87% accuracy, validated on historical data

Incident:

Local news investigation reveals approval rate for minority-owned businesses is 23% vs. 41% for non-minority businesses
Reporter requests explanation of algorithm and training data
Regulator opens investigation under fair lending laws

Questions for cross-functional team:

What went wrong? (Technical: fairness testing gaps)
What are we legally required to provide? (Legal: adverse action explanations)
What can we explain about the model? (Technical: interpretability limits)
What's our liability exposure? (Legal: potential penalties)
How do we fix it? (Technical: retraining, fairness constraints)
How do we prevent recurrence? (Governance: enhanced testing)
What do we tell customers? (Comms: transparency, remediation)
When can we redeploy? (Technical + Legal: validation + compliance) Working through this scenario together reveals translation gaps between technical and business language that separate training never surfaces.

Quarterly workshop cadence builds sustained literacy:

Q1: Model explainability and regulatory transparency requirements
Q2: Fairness testing and anti-discrimination law
Q3: Security, adversarial robustness, data protection
Q4: Incident response, crisis communication, remediation
Engineering implementation:

Python

literacy_assessment.py

class AILiteracyAssessment:
"""Track organizational AI literacy across roles"""

def __init__(self):
    self.role_competencies = {
        "executive": [
            "Understand strategic AI risks",
            "Interpret AI business cases",
            "Evaluate AI vendor claims",
            "Oversee AI governance"
        ],
        "manager": [
            "Identify appropriate AI use cases",
            "Set realistic AI expectations",
            "Manage AI-augmented teams",
            "Escalate AI concerns appropriately"
        ],
        "technical": [
            "Understand governance requirements",
            "Implement fairness constraints",
            "Document model limitations",
            "Conduct bias testing"
        ],
        "legal_compliance": [
            "Map AI to regulatory requirements",
            "Assess AI legal risks",
            "Draft AI-specific contract terms",
            "Conduct AI compliance audits"
        ]
    }

def assess_individual(self, role: str, employee_id: str) -> Dict:
    """Assess individual AI literacy"""
    competencies = self.role_competencies[role]

    assessment = {}
    for competency in competencies:
        # Assess through scenario-based questions
        score = self._assess_competency(employee_id, competency)
        assessment[competency] = score

    overall_score = np.mean(list(assessment.values()))

    return {
        "employee_id": employee_id,
        "role": role,
        "competency_scores": assessment,
        "overall_score": overall_score,
        "needs_training": overall_score < 0.7
    }

def identify_literacy_gaps(self, organization_assessments: List[Dict]) -> Dict:
    """Identify organizational literacy gaps requiring training"""
    df = pd.DataFrame(organization_assessments)

    # Gaps by role
    by_role = df.groupby('role')['overall_score'].mean()

    # Gaps by competency
    all_competencies = []
    for assessment in organization_assessments:
        for comp, score in assessment['competency_scores'].items():
            all_competencies.append({"competency": comp, "score": score})

    comp_df = pd.DataFrame(all_competencies)
    by_competency = comp_df.groupby('competency')['score'].mean()

    priority_training = by_competency[by_competency < 0.6].index.tolist()

    return {
        "literacy_by_role": by_role.to_dict(),
        "literacy_by_competency": by_competency.to_dict(),
        "priority_training_topics": priority_training,
        "overall_organizational_literacy": df['overall_score'].mean()
    }

Engineering primitive: The most effective AI literacy investment is cross-functional workshop sessions where technical and business teams work through real scenarios together. A workshop where a data scientist explains a model card to a compliance officer, who then explains regulatory requirements to the data scientist, produces more practical understanding than separate training courses. These workshops reveal translation gaps that cause miscommunication in daily operations.

Learn more about building comprehensive AI literacy programs →

Best Practice 10: Measure Business Value, Not Just Technical Performance
A governance framework that prevents every risk but blocks every value creation opportunity isn't serving the organization. Balance requires measuring both dimensions.

Balanced scorecard for AI systems:

Technical Performance Metrics Model accuracy: Precision, recall, F1-score, AUC on validation/test data Inference performance: Latency P50/P95/P99, throughput, resource utilization Reliability: Uptime, error rates, timeout frequencies
Business Impact Metrics Efficiency gains: Time saved, manual effort reduced, throughput increased Revenue impact: Conversion lift, customer lifetime value increase, pricing optimization Cost reduction: Process automation savings, error remediation cost reduction Customer satisfaction: NPS improvement, resolution time reduction, service quality scores
Risk and Compliance Metrics Fairness: Demographic parity, equalized odds across protected groups Security: Vulnerability scan results, penetration test findings, incident frequency Compliance: Audit findings, regulatory deficiencies, policy violations Explainability: Explanation availability, stakeholder comprehension scores
Adoption and Trust Metrics Usage rates: % of eligible decisions using AI, adoption by user segment Override rates: % of AI recommendations overridden by humans User satisfaction: Internal user NPS, feature request volume, support ticket trends Stakeholder trust: Executive confidence scores, board satisfaction with governance Engineering implementation:

Python

balanced_scorecard.py

from dataclasses import dataclass
from typing import Dict, List

@dataclass
class AISystemScorecard:
"""Balanced measurement across four dimensions"""

system_name: str
period: str  # e.g., "2024-Q1"

# Technical performance
technical_metrics: Dict[str, float]  # accuracy, latency, uptime

# Business impact
business_metrics: Dict[str, float]  # revenue, cost, efficiency

# Risk and compliance
risk_metrics: Dict[str, float]  # fairness, security, compliance

# Adoption and trust
adoption_metrics: Dict[str, float]  # usage, satisfaction, trust

def overall_health_score(self) -> Dict[str, float]:
    """Compute weighted health score across dimensions"""
    weights = {
        "technical": 0.25,
        "business": 0.35,
        "risk": 0.25,
        "adoption": 0.15
    }

    # Normalize each dimension to 0-1 scale
    technical_score = self._normalize_metrics(self.technical_metrics)
    business_score = self._normalize_metrics(self.business_metrics)
    risk_score = self._normalize_metrics(self.risk_metrics)
    adoption_score = self._normalize_metrics(self.adoption_metrics)

    overall = (
        weights["technical"] * technical_score +
        weights["business"] * business_score +
        weights["risk"] * risk_score +
        weights["adoption"] * adoption_score
    )

    return {
        "overall": overall,
        "technical": technical_score,
        "business": business_score,
        "risk": risk_score,
        "adoption": adoption_score
    }

def identify_weaknesses(self, threshold: float = 0.6) -> List[str]:
    """Identify dimensions scoring below threshold"""
    scores = self.overall_health_score()

    weaknesses = []
    for dimension, score in scores.items():
        if dimension != "overall" and score < threshold:
            weaknesses.append(f"{dimension} ({score:.2f})")

    return weaknesses

def generate_executive_summary(self) -> str:
    """Executive-friendly scorecard summary"""
    scores = self.overall_health_score()
    weaknesses = self.identify_weaknesses()

    summary = f"""
    AI System Health Report: {self.system_name}
    Period: {self.period}

    Overall Health: {scores['overall']:.1%}

    Dimension Scores:
    - Technical Performance: {scores['technical']:.1%}
    - Business Impact: {scores['business']:.1%}
    - Risk & Compliance: {scores['risk']:.1%}
    - Adoption & Trust: {scores['adoption']:.1%}
    """

    if weaknesses:
        summary += f"\nAreas Requiring Attention:\n"
        summary += "\n".join(f"- {w}" for w in weaknesses)

    # Business impact highlights
    summary += f"\n\nBusiness Impact This Period:\n"
    summary += f"- Revenue Impact: ${self.business_metrics.get('revenue_impact', 0):,.0f}\n"
    summary += f"- Cost Savings: ${self.business_metrics.get('cost_savings', 0):,.0f}\n"
    summary += f"- Efficiency Gain: {self.business_metrics.get('time_saved_hours', 0):,.0f} hours\n"

    return summary

ROI calculation framework:

Python

ai_roi_calculator.py

class AIProjectROI:
"""Calculate risk-adjusted ROI for AI investments"""

def __init__(self, project_name: str):
    self.project_name = project_name

def calculate_roi(self,
                 development_costs: float,
                 infrastructure_costs_annual: float,
                 operational_costs_annual: float,
                 revenue_impact_annual: float,
                 cost_savings_annual: float,
                 years: int = 3) -> Dict:
    """
    Calculate multi-year ROI

    Returns:
        Dict with NPV, IRR, payback period, ROI
    """
    # Total investment
    initial_investment = development_costs
    annual_costs = infrastructure_costs_annual + operational_costs_annual

    # Annual benefits
    annual_benefits = revenue_impact_annual + cost_savings_annual

    # Cash flows
    cash_flows = [-initial_investment]
    for year in range(1, years + 1):
        cash_flows.append(annual_benefits - annual_costs)

    # NPV (assuming 10% discount rate)
    discount_rate = 0.10
    npv = sum(cf / (1 + discount_rate)**i for i, cf in enumerate(cash_flows))

    # Simple ROI
    total_investment = initial_investment + (annual_costs * years)
    total_benefits = annual_benefits * years
    roi = (total_benefits - total_investment) / total_investment

    # Payback period
    cumulative = -initial_investment
    payback_period = None
    for year in range(1, years + 1):
        cumulative += (annual_benefits - annual_costs)
        if cumulative > 0 and payback_period is None:
            payback_period = year

    return {
        "npv": npv,
        "roi": roi,
        "payback_period_years": payback_period,
        "total_investment": total_investment,
        "total_benefits": total_benefits,
        "annual_net_benefit": annual_benefits - annual_costs
    }

Engineering primitive: Create balanced scorecards that track technical performance, business impact, risk metrics, and adoption rates. Review all four quadrants quarterly. A system scoring high in technical performance and compliance but low in business impact and adoption is a well-governed system that nobody uses—which means it's not delivering value. The balanced view prevents the pattern where technical teams celebrate model accuracy while business outcomes go unmeasured.

Conclusion: From Science Projects to Production Systems
The difference between AI projects that ship and AI projects that stall lies not in algorithm sophistication or model accuracy but in engineering discipline. Production AI systems require:

Governance with real authority over use case approval, deployment approval, and continuation decisions
MLOps infrastructure providing reproducibility, automation, and observability at scale
Risk-tiered lifecycle controls applying validation rigor proportional to potential harm
Modular, testable pipelines with automated quality gates catching regressions before production
Rigorous third-party AI management extending governance beyond organizational boundaries
Phased deployment with statistical validation catching problems when they're cheap to fix
Effective human oversight designed to function rather than satisfy compliance theater
Continuous drift monitoring with automated response workflows triggering investigation and retraining
Cross-functional literacy building shared understanding that enables collaboration
Balanced measurement tracking business value alongside technical performance and risk metrics

Organizations that manage AI projects like software projects—fixed requirements, linear development, deploy-and-forget operations—produce systems that work in notebooks and fail in production. The model drifts without detection. Governance exists without function. Business cases remain unverified because nobody measured outcomes.

Organizations that apply AI-specific engineering practices build production systems that deliver sustained value. Models get developed with statistical rigor. Deployment happens with proper monitoring. Maintenance continues with disciplined retraining. Measurement validates business impact.

An AI project managed for its first 30 days produces a demo. An AI project managed for its full lifecycle produces durable business value.

Which of these ten practices is weakest in your current AI engineering approach? Fix that before your next deployment.

About the Author
The frameworks, tools, and implementation guidance in this article come from Prof. Hernan Huwyler's applied research and consulting work. Prof. Huwyler, MBA, CPA, CAIO serves as AI GRC Consultancy Director, AI Risk Manager, and Quantitative Risk Lead, working with organizations across financial services, technology, healthcare, and public sector to build practical AI governance frameworks that survive production deployment and regulatory scrutiny.

His work bridges academic AI risk theory with the operational controls organizations actually need to deploy AI responsibly. As Speaker, Corporate Trainer, and Executive Advisor, he delivers programs on AI compliance, quantitative risk modeling, predictive risk automation, and AI audit readiness for executive teams, boards, and technical practitioners.

His teaching and advisory work spans IE Law School Executive Education and corporate engagements across Europe. Based in Copenhagen Metropolitan Area, Denmark, with professional presence in Zurich and Geneva, Switzerland, Madrid, Spain, and Berlin, Germany.

Code repositories, risk model templates, and Python-based tools for AI governance:
https://hwyler.github.io/hwyler/

Ongoing writing on Governance, Risk Management and Compliance:
https://mydailyexecutive.blogspot.com/

AI Governance technical blog:
https://hernanhuwyler.wordpress.com

Connect on LinkedIn:
linkedin.com/in/hernanwyler

If you're building production AI systems, establishing MLOps infrastructure, or preparing for regulatory compliance requirements, these materials are freely available for use, adaptation, and redistribution. The only ask is proper attribution.

Why I Write About AI Governance (And Why It Actually Matters)

Hernan Huwyler — Mon, 13 Apr 2026 21:16:08 +0000

I have spent the last two decades sitting in rooms where smart people make expensive mistakes with technology they do not fully understand.

I have watched boards approve AI initiatives without asking basic questions about data lineage, monitoring, and accountability.

I have seen compliance teams try to retrofit controls onto systems that were already in production, with customers already affected.

I have also debugged Monte Carlo risk models at 2 AM because someone assumed “AI risk” was just another flavor of traditional IT risk.

This blog exists because I got tired of watching the same failures repeat.

Most AI governance content falls into two categories that do not help you when the pressure is real.

It is either academic work that never reaches the operating model, or vendor content that sounds confident but collapses when you ask, “What evidence would an auditor accept?”

I write for the person who has to defend decisions, not just describe them.

If you are the risk manager who just inherited AI oversight with zero training, I know what that feels like.

If you are the compliance officer trying to determine whether the EU AI Act applies to your “simple chatbot,” I have been in that conversation.

If you are an internal auditor asked to validate a machine learning model and you do not know Python, you are not alone.

If you are a Chief AI Officer hired to “govern AI responsibly” but given no budget and a six‑month deadline, you have a structural problem, not a motivation problem.

If you need practical frameworks that survive contact with reality, not aspirational principles that fall apart under audit, you are in the right place.

What I mean by “AI governance” (in plain terms)

I do not treat AI governance as an ethics essay.

I treat it as the operating system that makes AI systems deployable, auditable, and recoverable.

In practice, that means answering questions like these with evidence:

Who owns this AI system in production, and who can pause it?

What data trained it, and what data is it using today?

What controls stop it from leaking confidential information?

How do we detect model drift, performance decay, bias shifts, or unsafe behavior after release?

What is the incident playbook when it fails at scale?

If you cannot answer those questions, you do not have governance. You have activity.

Where AI governance collides with AI development

AI systems do not fail like traditional software.

Software is mostly deterministic. You ship code, it behaves as written.

AI systems are probabilistic and data-dependent. You ship code plus a model plus a moving data environment, and behavior changes even when the code stays the same.

That is why “approval at launch” is weak control design.

In the real world, governance has to plug into the AI delivery pipeline, not sit beside it.

Here is the lifecycle I anchor most programs on:

Data → Training → Validation → Deployment → Monitoring → Change control → Retirement

If your controls only exist at “Validation,” you will miss most failures that occur after deployment.

Common failure patterns I keep seeing (and why they are expensive)

Teams build a model that performs well in a notebook, then discover they have no ModelOps or MLOps path to deploy it safely.

Monitoring is limited to uptime and latency, while the real risk is silent performance degradation, drift, or a shift in user behavior.

Third-party AI is onboarded through procurement as if it were a normal SaaS tool, without vendor evaluation on training data use, model change notifications, or audit rights.

Controls exist as documents, but they are not enforced by pipelines. No gating tests, no versioning discipline, no evidence trail.

The organization cannot produce an inventory of AI systems in production, so it cannot manage what it cannot see.

What you will actually find here

This is not a blog about “trust” as a slogan.

It is a working notebook of governance mechanisms that hold up under executive pressure, regulatory scrutiny, and operational incidents.

You will find implementation guidance that assumes real constraints: limited budget, skeptical stakeholders, legacy systems, and teams who want to ship.

You will also find technical content that bridges governance with development practices, including monitoring, testing, validation, and evidence generation.

In particular, I publish:

Practical implementation guides for standards such as ISO/IEC 42001, ISO/IEC 23894, and EU AI Act aligned governance approaches.

Quantitative risk models in Python and R that translate “this might be biased” into “this is the probable financial exposure under defined scenarios.”

Failure stories from real projects, including the controls that did not work, the assumptions that were wrong, and the fixes that survived audit and remediation cycles.

My bias as a practitioner

I am slightly impatient with governance that cannot be tested.

If a control cannot produce evidence, it is not a control. It is a sentence.

If a policy cannot be operationalized into build gates, monitoring checks, and incident routines, it is not governance. It is shelf decoration.

That is the perspective behind everything I publish.

A technical example of what “governance in the pipeline” looks like

When I say governance should be real, I mean it should show up in the same places your engineers already work.

For example, a release gate that blocks deployment if minimum evidence is missing:

release_gates:
  - name: model_card_required
    rule: "model_card.exists == true"

  - name: monitoring_required
    rule: "monitoring.drift.enabled == true AND monitoring.performance.enabled == true"

  - name: high_risk_extra_checks
    rule: "if risk_tier == 'high' then fairness_test.passed == true AND human_override.enabled == true"

This is not about bureaucracy.

This is about preventing the most common enterprise failure mode: shipping an AI system that nobody can explain, monitor, or shut down safely.

Published articles and practical guides

Below is a curated index of articles. Each one is designed to solve a specific friction point I keep seeing in enterprise AI.

If you are time-poor, skip to the domain that matches your current pain.

AI governance frameworks and standards

Practical ISO/IEC 42001 Implementation Guide

A step-by-step approach to implementing an AI Management System. I focus on governance structure, control design, documentation, audit readiness, and how to integrate this with existing GRC.

How to Actually Use ISO/IEC 23894 for AI Risk Management

A practical playbook for operationalizing AI risk management. Less philosophy, more workflow, scenario libraries, and monitoring expectations.

A 12-Step Procedure Merging ISO 27005, ISO 23894, ISO 42001, and FAIR

An integrated risk method that teams can execute without turning the process into a six-month consulting project.

Implementation Tips for ISO/IEC 42005 AI Impact Assessments

How to run impact assessments that produce usable outputs: stakeholder mapping, scoring, mitigations, and documentation that stands up in review.

Practical Implementation Tips for AI Project Alignment

How to align AI work with strategy and risk appetite so you do not end up with technically strong projects that deliver weak enterprise value.

Chief AI Officer (CAIO) operating model and accountability

What a Chief AI Officer Actually Owns, and What Should Stay With Risk, Legal, and IT

A practical CAIO responsibility map across governance, operational assurance, organizational enablement, and strategic influence, aligned to three lines of defense.

AI risk assessment and quantification

AI Risk Modeling: Beyond “Is AI Accurate?”

How I quantify AI exposure using frequency-severity logic, scenario analysis, and loss distributions, then connect it to board-level risk language.

The AI Risk Taxonomy Most Organizations Never Build

A taxonomy approach that prevents the “one heat map to rule them all” problem.

The AI Loss Taxonomy Your Risk Assessments Are Missing

A structured way to think about loss: direct financial, regulatory, litigation, reputational, churn, and operational disruption.

Practical AI Assessments: Risk, Impact, and Feasibility

A combined assessment workflow that produces a decision, not just a report.

Implementation Tips for Expert Calibration and AI-Augmented Risk Estimation

How to reduce “confident guessing” in risk scoring and produce estimates you can defend.

AI security, threat modeling, and red teaming

The 45 AI Threat Vectors Your Security Team Probably Isn’t Tracking

A threat taxonomy that includes data poisoning, model extraction, prompt injection, membership inference, backdoors, and supply chain risks.

AI Threat and Vulnerability Assessment Framework

A structured approach to AI threat modeling and vulnerability assessment, designed to be run repeatedly, not once.

Practical AI Red Team Implementation Tips for Safer, More Resilient AI Systems

How to stand up an AI red team, what scenarios to test, how to document results, and how to drive remediation that actually sticks.

Guide to AI Agent Risk and Control Management Across the Full Lifecycle

Agents raise the stakes because they can take actions, not just generate text. This guide focuses on delegation limits, human-in-the-loop design, monitoring, and liability.

Quantitative risk modeling and predictive analytics

Quantitative Risk Assessment Using Monte Carlo Simulations and Convolution Methods in R

Executable methods for compound loss modeling, loss exceedance curves, reserves, and sensitivity analysis.

Machine Learning for Advanced Predictive Risk Modeling

How to use supervised learning for risk prediction responsibly, including validation and explainability.

Predictive Risk Model That Makes the Fewest Expensive Mistakes

Cost-sensitive modeling. Because accuracy is rarely the business objective.

How to Explain AI Risk Models So Regulators Actually Trust Them

A communication framework for regulators, auditors, and boards, anchored in assumptions, sensitivity, limitations, and evidence.

AI project management and delivery (where good ideas die)

Field Guide to the 8 Factors That Determine Success or Failure of AI Projects

A practical view of why AI programs succeed or stall: sponsorship, data maturity, team design, and operating model.

Practical Fixes for Why Data Science Projects Fail

Root causes and fixes that reduce rework and prevent “pilot purgatory.”

Managing AI Development and Deployment Projects

A disciplined approach that respects the exploration phase but still gets to production with control.

Managing AI Projects with Agile Exploration and MLOps

How I combine experimentation with release discipline so governance does not become the enemy of shipping.

How to Build the Right AI Delivery Team

Roles, responsibilities, and why missing a single capability (like platform engineering or domain expertise) can break delivery.

Why Separating Your AI Build Team from Your AI Ops Team Guarantees Failure

An organizational design problem disguised as a tooling problem.

Resource Estimation for AI Projects

A reality-based way to estimate compute, people, data effort, and vendor spend.

Goal Setting for AI Projects

How to set measurable AI goals that include constraints, not just targets.

Feasibility Assessment for AI Projects

Technical feasibility, economic feasibility, operational feasibility, and regulatory feasibility, evaluated upfront.

AI monitoring, validation, and maintenance (where governance becomes real)

Model Selection and Validation for AI Projects

How to choose models and prove they generalize, including cross-validation and holdout discipline.

The Model Robustness and Monitoring Playbook

Drift detection, degradation triggers, and what to monitor beyond accuracy.

Practical Monitoring and Evaluation for AI Projects

A full monitoring architecture: technical metrics, model metrics, business metrics, and governance metrics.

Practical KPI Tracking for AI Projects

Leading and lagging indicators that let you intervene before failure becomes visible to customers.

Practical Post-Deployment Maintenance for AI Systems

Versioning, retraining cadence, dependency updates, security patching, and retirement discipline.

AI Deployment Governance for Feedback Loops and MLOps

Controls for the feedback loop so you can improve systems without creating uncontrolled change risk.

Spent 5 Years Validating Enterprise AI Models: Here’s What I Learned

Common validation failures, regulator expectations, documentation patterns, and what breaks most often in production.

How to use this index (fast)

If you are building an AI governance program from scratch, start with ISO/IEC 42001 and the CAIO responsibilities map, then move into monitoring and incident readiness.

If you are preparing for audit or regulatory scrutiny, focus on evidence artifacts: inventory, model documentation, monitoring records, change logs, and vendor governance.

If you are a technical lead trying to ship responsibly, start with the MLOps governance, monitoring, and security testing articles. That is where most “surprises” hide.

Closing

I do not write to sound smart.

I write because AI governance fails quietly until it fails loudly, and by then, the people in risk, compliance, and audit are the ones asked to explain what happened.

If you want a specific topic covered next, tell me what you are being asked to govern this quarter: customer-facing models, internal copilots, vendor AI, or autonomous agents.

AI Policy, Compliance, and Regulatory Frameworks
Responsible AI Policy Categories and Implementation Framework
https://hernanhuwyler.wordpress.com/responsible-ai-policy-categories/
Taxonomy of responsible AI policies covering ethics, fairness, transparency, accountability, privacy, security, safety, and human oversight. Includes policy templates, implementation checklists, training programs, and compliance verification protocols.
Rules for AI Use: Accountability, BYOAI, Safety by Design, and Content Provenance
https://hernanhuwyler.wordpress.com/rules-for-ai-use-accountability-byoai-safety-by-design-and-content-provenance/
Corporate policy framework governing employee AI usage including bring-your-own-AI (BYOAI) protocols, accountability assignments, safety-by-design requirements, and content provenance tracking for generative AI outputs.
Practical CAIO Responsibilities: What Chief AI Officers Actually Do
https://hernanhuwyler.wordpress.com/practical-caio-responsibilities/
Role definition for Chief AI Officer positions including strategic responsibilities (AI roadmap, portfolio governance), operational responsibilities (project oversight, resource allocation), and assurance responsibilities (risk management, regulatory compliance, board reporting).
Compliance Controls for AI Systems
https://hernanhuwyler.wordpress.com/compliance-controls-for-ai/
Control catalog mapping AI-specific compliance requirements to implementable controls across data governance, model development, deployment, monitoring, and documentation domains. Aligned with EU AI Act, GDPR, sector-specific regulations.
Practical Implementation Tips for Building and Maintaining an AI Compliance Register
https://hernanhuwyler.wordpress.com/practical-implementation-tips-for-building-and-maintaining-an-ai-compliance-register/
Operational guidance for constructing AI compliance registers tracking regulatory obligations, control mappings, evidence collection, audit trails, and compliance status reporting across multiple jurisdictions.
Practical Implementation Tips for AI Fundamental Rights Taxonomy
https://hernanhuwyler.wordpress.com/practical-implementation-tips-for-an-ai-fundamental-rights-taxonomy/
Framework for identifying and assessing fundamental rights impacts of AI systems as required by EU AI Act. Covers rights taxonomy, impact assessment methodologies, mitigation planning, and stakeholder consultation protocols.
Practical Implementation Tips for Fundamental Rights Impact Assessment for High-Risk AI Systems
https://hernanhuwyler.wordpress.com/practical-implementation-tips-for-a-fundamental-rights-impact-assessment-for-high-risk-ai-systems/
Step-by-step procedure for conducting fundamental rights impact assessments (FRIA) for high-risk AI systems under EU AI Act Article 27. Includes assessment templates, stakeholder engagement protocols, impact scoring, mitigation planning, and documentation requirements.
Modeling Practices for Regulated AI Systems
https://hernanhuwyler.wordpress.com/modeling-practices-for-regulated-ai/
Best practices for developing AI models in regulated industries (financial services, healthcare, critical infrastructure) covering model governance, validation standards, documentation requirements, change control, and regulatory submission protocols.

AI Procurement and Vendor Management
AI Procurement Controls and Vendor Risk Management
https://hernanhuwyler.wordpress.com/ai-procurement-controls/
Comprehensive framework for procuring AI systems and services including vendor assessment criteria, technical due diligence protocols, contractual protections, service level agreements, audit rights, data handling requirements, and ongoing vendor monitoring.
How to Negotiate AI Agreements That Protect Data, Value, and Liability
https://hernanhuwyler.wordpress.com/how-to-negotiate-ai-agreements-that-protect-data-value-and-liability/
Legal and commercial negotiation strategies for AI vendor contracts covering intellectual property rights, data ownership, model performance warranties, liability caps, indemnification clauses, termination rights, and regulatory compliance responsibilities.