Most advice about building with AI is written for a world that does not apply to banks, hospitals, insurers, or anyone who answers to a regulator. The advice says move fast, ship the rough version, learn from real users, iterate in production. Inside a regulated estate, that advice is not just wrong, it is the fastest route to a fine, a halted rollout, or a regulator who now reads everything you ship twice.
The instinct this breaks is the one every good AI team has been trained to trust. In an unregulated product, speed is the dominant strategy: the team that ships and learns beats the team that deliberates. In a regulated estate, the cost of being wrong in production is not a bad week, it is a reportable event, a customer who was treated unfairly, a decision you cannot explain to an examiner. The arithmetic of speed inverts, and most teams do not notice until they have already built the thing the wrong way.
This is not an argument for doing less with AI in regulated industries. It is an argument that the hard part is different, and the teams that win are the ones that understood the difference before they started building rather than after their first audit.
The rule that breaks the playbook
The standard AI playbook rests on a quiet assumption: that you can put an imperfect system in front of real users, watch what happens, and improve it. That assumption is what regulation removes. A lending model that is right most of the time is not a promising first version, it is a system that treats some real applicants unfairly on days you cannot predict, and in a regulated estate that is the thing the rules exist to prevent.
So the loop that makes AI teams fast, ship, observe, correct, is exactly the loop a regulator will not let you run on live customers. You cannot learn your fairness properties from production incidents. You cannot discover your failure modes by letting them happen to people who have legal protections. The learning has to move upstream, before deployment, into evidence you can produce on demand, and that single shift reshapes everything about how the work is sequenced.
The teams that struggle are the ones that treat compliance as a gate at the end: build the model the usual way, then hand it to a risk function to bless. The risk function cannot bless what it cannot inspect, the model was never built to be inspected, and the project stalls in a standoff that burns months. The teams that move are the ones that built the evidence as they built the system, because they understood from the first week that in a regulated estate the model and its justification are the same deliverable.
Why regulated is a different kind of hard
It helps to be precise about what regulation actually demands, because the word covers several different constraints that are easy to blur together.
The first is explainability: for certain decisions, you must be able to say why, in terms a human can understand and a customer can challenge. The second is fairness: you must be able to show the system does not discriminate against protected groups, and show it with evidence, not assurances. The third is auditability: you must be able to reconstruct, after the fact, exactly what the system did and why, for any individual decision, sometimes years later. The fourth is accountability: a named human must own the model's behaviour, which means that human needs to genuinely understand it, not just sign for it. And the fifth is often data residency and control: where the data lives, who can see it, and what can be sent to a third party, including the provider of a model you are renting.
Each of these is a design constraint that must be present from the start, because none of them can be bolted on afterwards. You cannot make an unexplainable model explainable by adding a report. You cannot make an unauditable system auditable by writing documentation about it later. The constraint has to live in the architecture, and architecture is the one thing that is expensive to change once the system exists. That is why regulated AI is not normal AI plus paperwork. It is a different engineering problem whose hardest requirements are fixed before the first model is trained.
Explainability is a product requirement, not a nice-to-have
In most AI products, explainability is a feature you might add for user trust. In a regulated estate, it is frequently a legal requirement, and that changes which systems you are even allowed to build.
This is where many teams make their most expensive mistake. They reach for the most accurate model available, a large opaque one, get a result that beats the simpler option by a few points, and only later discover they cannot deploy it because they cannot explain its decisions to the standard the regulation requires. The accuracy they bought is worthless if the system is not permitted, and a slightly less accurate model that can be explained and defended is not a compromise, it is the only one that ships.
So in a regulated context, explainability sits alongside accuracy as a first-class selection criterion, not a tie-breaker. Sometimes that means choosing an inherently interpretable model over a black box that scores higher. Sometimes it means wrapping a complex model in constraints and checks that make its behaviour defensible. Sometimes it means the right answer is not a model at all, but a transparent rule a customer and an examiner can both follow. The question is never only "which is most accurate." It is "which is most accurate among the systems we can actually stand behind when someone asks us to justify a decision," and a team that skips the second half of that sentence builds something it cannot use.
The model is a decision you must be able to defend
The deepest shift is that in a regulated estate you are not shipping a model, you are shipping a decision that a specific person is accountable for and that a specific customer can contest. The model is just the mechanism. What the regulation cares about is the decision and whether it can be justified, which means the thing you are really building is a defensible decision process that happens to use a model inside it.
That reframing changes what "done" means. A model that performs well on a test set is not done. A decision process is done when a named owner understands it well enough to defend it, when any individual output can be reconstructed and explained, when the fairness properties have been measured and documented, and when there is a clear, tested path for a customer to challenge a decision and have it reviewed. All of that is the deliverable, and the model is perhaps a quarter of the work.
Teams that internalise this stop asking "is the model accurate enough" as the final question and start asking "can we defend every decision this will make, to the person it affects and to the regulator who protects them." The second question is harder, it is the right one, and it is the one that was always going to be asked. Asking it first is the difference between building the system once and building it twice.
Build the audit trail before you build the feature
The practical discipline that separates teams who ship in regulated estates from teams who stall is counterintuitive: build the audit trail first, before the feature works, not after.
The reason is that an audit trail added later is never complete. You will always have failed to capture some input, some version, some intermediate decision, that an examiner later asks for and you cannot produce, because the system was not recording it at the moment that mattered and that moment has passed. In a regulated estate the ability to reconstruct a decision is not a reporting feature, it is a precondition for being allowed to make the decision at all, so it has to exist before the first real decision is made.
Concretely, that means logging the complete chain from the start: which model version, which inputs, which data it was trained on, which prompt or configuration, which output, which human reviewed it, and what happened next, all captured as the system runs, all queryable for any individual case, all retained for as long as the rules require. It feels like overhead when the feature is not even working yet. It is not overhead. It is the part of the system that makes the feature legal, and a team that treats it as something to add once the interesting work is done has inverted which part was actually interesting.
Humans in the loop, by design and by law
In an unregulated product, how much to automate is an optimisation question: automate where it is cheaper and good enough, keep a human where it is not. In a regulated estate, the human in the loop is frequently a requirement, and the design question becomes how to make that human genuinely effective rather than a rubber stamp.
This is subtle, because a human in the loop who approves whatever the model suggests, because the model is usually right and checking is tedious, is worse than no human at all: they add cost, they add delay, and they add the illusion of oversight without its substance, which is exactly what an examiner is trained to detect. The requirement is not a human present, it is a human in genuine control, and those are very different systems to build.
Making the human effective means designing the handoff deliberately. The system has to surface why it reached its conclusion, not just what the conclusion is, so the reviewer has something real to evaluate. It has to flag its own uncertainty, so attention goes where it is needed rather than being spread evenly across cases that do not need it. It has to make disagreeing with the model as easy as agreeing, so the path of least resistance is not blind approval. The goal is a reviewer who catches the cases the model gets wrong, and that is a product you design on purpose, not a checkbox you add because the rules demand a name in a field.
The advantage hiding inside the constraint
It is tempting to read all of this as a list of reasons regulated industries move slowly with AI, and to treat the constraint as pure cost. That reading misses where the advantage is.
The constraints that regulation imposes, explainability, auditability, measured fairness, genuine human oversight, a named owner who understands the system, are, with few exceptions, the properties of a well-engineered AI system in any industry. The regulated estate is simply forced to build them, where an unregulated competitor is merely advised to and usually does not. The discipline that feels like a tax is the same discipline that produces systems people can actually trust, that fail in understood ways, that can be improved because they can be inspected. A team that masters it in a regulated context has built a capability that travels.
So the frame that serves a leader is not "how do we do AI despite the regulation." It is "the regulation is forcing us to build AI properly, and the muscle we develop doing so is a durable advantage over everyone who got to cut the corners we could not." The walls are real, moving fast inside them is genuinely not an option, and the teams that accept that early and build for defensibility from the first week do not just satisfy the regulator. They end up with the AI that works, because defensible and reliable turn out to be the same system seen from two directions.
I write a field-note series on shipping AI that actually works in production. The original of this piece, and the rest, live on my LinkedIn. Originally published on LinkedIn.
Top comments (0)