Track one number when you put a generative model into a review workflow: how often reviewers accept the draft unchanged.
Above roughly 95% and they've stopped reading. That's automation bias, and it looks exactly like success on every dashboard you have. The person's job changed from doing the work to approving it, which is a different job with different failure modes, and the main one is approving what they should have caught.
This is the shape of the whole problem. A prototype proves the model can produce the artifact. Production asks whether you can show a regulator, six months later, why a particular output was produced — and whether a customer can challenge it. Those are design problems long before they're engineering problems.
The integration work is roughly 20% model and 80% everything else: data pipelines, evals, audit logging, human review paths, and the interface showing a user what the model did.
Three things change when the output is probabilistic
The same input can produce a different artifact tomorrow. Interfaces built on deterministic assumptions break quietly. Pin temperature and seed or you cannot reproduce a decision.
The audit surface grows. Every inference becomes a record you may need to reconstruct.
The human role moves from producing to approving, with the acceptance-rate problem above.
Generative and predictive are different products wearing one name
A credit model returning a default probability is predictive. An assistant drafting the adverse-action notice that follows is generative. Vendors blur this and buyers inherit the blur — one platform sells "AI for lending" and you've bought two unrelated risk profiles under one contract.
Predictive Generative
Output Score, class, ranking Text, document, summary
Evaluation Precision, recall, AUC vs labels Faithfulness, groundedness, human preference
Failure mode Systematic bias against a group Confident fabrication, one output at a time
Found by Aggregate analysis Per-instance review
Reproducibility Deterministic given version Varies unless parameters pinned
Cost profile Training-heavy, inference-cheap Training-light, inference-expensive
A generative feature needs a review path. A predictive feature needs a challenger model. Building one and governing it as the other is the most expensive mistake in the category, because it's usually found during an examination rather than during a sprint.
RAG sits between them and confuses things further. Govern the retrieval as a data-access control and the generation as a content control. Treating it as one component makes both halves harder to defend.
Four prerequisites, each with a pass/fail test
Run these in week one, on paper, before anyone writes a prompt:
Data — can you pull 200 real cases with labeled correct outcomes without a manual export? If that takes more than two days, the data problem is your project.
Team — is a compliance reviewer named in the sprint, not on the distribution list?
Baseline — is there a current metric with today's number written next to it?
Sponsor — does one executive own the outcome and the budget, in writing?
The data test fails most often, and it fails in week five, after the model work is already scheduled. That moves the launch by a quarter.
Write the eval suite before picking the model
The sequence teams actually run: pick the model in week one because it's the interesting decision, design the interface around whatever it produces, hand the whole thing to compliance in week ten. Reversing that is most of what separates a six-week build from a six-month one.
The dependency order that can't move:
You cannot validate a model against evals you haven't written.
You cannot write evals without knowing what a correct output looks like in the workflow.
So the domain expert who does this work today has to be in the room before model selection, scoring 100 to 300 real cases.
Cost, latency, data residency, and vendor retention policy narrow the field faster than benchmark scores do. And test for disparate outcomes on protected characteristics even when the output is a draft rather than a decision — a drafting assistant that writes warmer letters to one group is a fair-lending problem wearing a different hat.
Design the reviewer's screen before the customer's
The review interface decides whether reviewers catch errors or rubber-stamp them, and that single behaviour sets the feature's real error rate more than model quality does. It's the part of the work that never shows up in a portfolio.
Beyond acceptance rate, two things help: put the retrieved records next to the generated draft so the grounding is visible, and seed known-bad cases into the queue to check people are still catching them.
On the customer side, three patterns carry the trust load:
Confidence disclosure that states the basis, not a number. "Drafted from your March statement and two prior disputes" tells a user more than "87% confidence" and is much harder to misread.
Reversible action. Anything the model initiates gets an undo with a stated window. Where reversal is impossible — a sent payment — require confirmation before rather than explanation after.
A visible human path. One click, with expected response time on the button.
When a fraud hold lands on a legitimate transaction, the customer doesn't want an explanation of the model. They want the hold lifted and they want to know how long that takes. Trust in financial products is mostly a function of recoverability. Users trust what they can undo more than what they can read.
Two anti-patterns: acting without confirmation on anything material, and decorative explainability — a "why did I see this" link returning generic text. A weak explanation is worse than none, because it signals the team knew the answer mattered and chose not to give one.
Audit log before prompt
The artifact people underbuild. It has to store the prompt, the retrieved context, the model version, the parameters, the output, and the reviewer's decision — keyed so one case can be reconstructed on request.
The examiner's first question is rarely about the model. It's who signed off, what they saw when they did, and whether you can produce that record now. Teams that treat compliance as a document written after the build find the document has to describe decisions nobody wrote down.
Cheap to build in week one. Expensive to retrofit, and the feature sits idle while it happens.
What it costs, and where
A first production feature in a regulated fintech workflow: roughly $110,000 to $220,000 for a customer-support-style assistant, $320,000 to $1.1M for transaction monitoring, over three to six months. Run costs $2,000 to $25,000 a month depending on volume.
The model is the cheap part — inference on a well-scoped workflow is often under $2,000 a month at pilot volume. The spread comes from data readiness and regulatory tier, not model choice. Rough split: data prep 25–35%, model integration and evals 15–20%, interface and review design 20–25%, audit logging 10–15%, validation and documentation 10–20%.
What pushes a build past the top of the range is nearly always the same thing: the workflow wasn't scoped to one queue, and it grew a second and third use case before the first one shipped.
One trade-off this approach doesn't solve. Running compliance in parallel from step one makes your first feature slower than a team that skips it — measurably, often four to six weeks. The payback lands on features two and three, when the control framework is reusable. If your organization judges the first project on speed alone, negotiate the measure before the project starts rather than cutting the governance work.
Scope to one workflow with a number attached. Write the kill criteria before the code.
Full guide — the seven-step process, the regulatory table, and build vs buy: see the full guide
Written by Bohdan Varshchuk, CTO at Teamvoy. More engineering writing at teamvoy.com/blog.
Top comments (0)