Most AI governance lands on engineering teams as a document. Someone writes an acceptable use policy, it is circulated, and nothing about how anyone works actually changes. Six months later a customer asks which of their data has been sent to a model provider and the policy turns out not to answer that.
Governance only holds when it is a property of the workflow rather than a file in the wiki. Below is what that looks like in practice for a team that ships software, with the controls expressed as things that happen in a repository rather than things people are supposed to remember.
None of this requires a certification programme. It happens to line up with the standard frameworks, which is convenient later, but the reason to do it is that it answers questions you will be asked.
1. Classify the data before it reaches a provider, in code
The single most useful control is also the least glamorous: know what you are sending.
If your application sends user content to a model API, that call site is a trust boundary and should be treated like one. In practice:
Put model calls behind one internal client rather than calling the provider SDK from twelve places. That gives you exactly one place to add redaction, logging and limits, and one place to change providers.
Make the caller declare the data class. A required argument, not a convention: internal, customer content, or restricted. Reject restricted at the boundary unless there is an explicit allowance.
Strip what you do not need before the call. Identifiers, emails, and free-text fields that nobody asked the model to reason about are the usual offenders.
The test for this control is simple. Someone should be able to answer "what categories of data has this product sent to a model provider?" by reading one module, not by interviewing the team.
2. Log the prompt, the response, and the version
Treat inference like any other external call you would be embarrassed not to have logged.
At minimum, record the model and version, the timestamp, the caller, a hash of the input, and whether the output was used or discarded. Whether you store the full prompt text depends on what is in it, which is exactly why step one comes first.
The reason is not compliance theatre. When a model changes behaviour after a provider updates a version, this log is the only way to tell whether your output changed with it. Without it you are debugging a system whose behaviour is not reproducible and whose history you did not keep.
3. Pin versions and treat a model change as a deploy
A model identifier that points at "latest" is an unreviewed dependency update in production, applied by somebody else, on their schedule.
Pin to an explicit version. Put the upgrade through the same path as any other dependency bump: a change, a review, a test run, a deploy you can roll back. Providers deprecate versions, so track the deprecation dates somewhere that will interrupt you before the cutoff rather than after.
4. Make evaluations part of the test suite
If the output of a model affects what a user sees, you need a regression test for it, and "we tried it and it looked good" is not one.
Keep a small set of fixed inputs with expected properties, and assert on properties rather than exact strings: refusals happen where they should, required fields are present, outputs stay inside length and format bounds, and known bad inputs do not produce the bad output they used to. Run it in CI on every change to the prompt, the model version, or the surrounding code.
This is the control that catches the prompt edit that quietly broke the thing nobody was looking at. It is also, not incidentally, the evidence an assessor wants when they ask how you know the system performs as intended.
5. Put the AI question in the pull request template
Three checkboxes, applied only when a change touches a model call:
Does this change what data is sent to a provider? Does this change the model, the version, or the provider? Does this output reach a user without a human in between?
Any yes routes to a short review. Nothing else does. The reason this works where a policy document does not is that it appears at the moment the decision is being made, by the person making it, and takes fifteen seconds when the answer is no.
6. Decide where a human belongs, and write it down once
"Human in the loop" is used as though it means one thing. It needs to be specific per feature, and the honest version is a sentence in the code review, not a principle in a policy.
The useful distinction is what the output does. Output that informs a person who then decides needs little. Output that is shown to a customer as though authoritative needs review of the failure mode, not of every instance. Output that triggers an action, sends a message, moves money, changes access, needs a person or a hard constraint in front of it, every time.
Write which of those three applies next to the feature. When it changes, the pull request template in step five catches it.
7. Give agents the same scrutiny as a new employee
An agent with a token is a principal in your system, and the questions are the ones you would ask about any other account.
What can it reach, and is that scope the minimum? Whose credential is it using, and is it a service identity rather than a person's token? What is the revocation path, and has anyone tested it? Is its activity attributable in your logs, or does it appear as the human whose token it borrowed?
That last one is the common failure, and it is the one that makes an incident unreconstructable. An agent acting under a person's credential means your audit trail is wrong about who did what.
8. Keep the register short enough to stay true
One row per AI-enabled feature: what it does, which model and version, what data it sees, where the human sits, and who owns it. Five columns, in version control, next to the code.
A register that is complete and shallow beats a thorough one covering a third of your systems, because the gap in the thorough one is invisible. If you want a reference for the fuller control set this maps onto, IRM's free ISO 42001 gap assessment walks Clauses 4 to 10 and the 38 Annex A controls, which is a reasonable map of what a mature version of this looks like.
About IRM's AI Governance services
Disclosure first, so you can weigh the advice accordingly: this post comes from IRM Consulting & Advisory, and the links in it point to our own pages.
The gap this post is about is the one we see most: governance written as policy for a company that builds software, with nothing connecting it to the pull request where the decisions actually get made. The fix is rarely more documentation. It is usually a handful of controls relocated into the workflow, which is cheaper and holds better.
IRM Consulting & Advisory provides AI governance and ISO 42001 consulting to SaaS and AI-first companies in Canada and the United States, covering AIMS scoping, AI risk assessments and the Annex A controls.
One question for anyone further along than this: have you found a way to version prompts that your team actually sticks to? Everyone agrees they should be versioned and almost nobody treats them like code.
Top comments (0)