Originally published at twarx.com - read the full interactive version there.
Last Updated: July 7, 2026
Most AI technology workflows in financial services are solving the wrong problem entirely.
The debate over custom Small Language Models (SLMs) versus off-the-shelf LLMs like GPT-4o or Claude 3.7 is dominating boardrooms in regulated finance right now — accelerated by McKinsey's newly published 'Five hallmarks of effective AI strategies in banking.' But the smartest teams have realized that the AI technology decision isn't really about the model at all. This is fundamentally a coordination problem, not a model-selection problem. The tools in play are real: fine-tuned SLMs on Anthropic and open-weight Llama derivatives, RAG pipelines on vector databases, and LangGraph orchestration layers.
By the end of this article you'll have a decision framework — the AI Coordination Gap — that tells you exactly what to deploy, what it costs, and where it breaks.
A production financial-services stack showing where a fine-tuned SLM and an off-the-shelf LLM each sit — and why the handoff between them is the real risk surface. This is the heart of the AI Coordination Gap.
Overview: Why the SLM-vs-LLM Question Is the Wrong Starting Point
Here's the counterintuitive truth that senior operators in banking are only now internalizing: the model is not your bottleneck. Coordination is.
When a Tier-1 bank benchmarks a fine-tuned 7B-parameter SLM against GPT-4o for, say, transaction dispute triage, the accuracy delta is often smaller than 4 percentage points on well-scoped tasks. Yet the total cost of ownership, latency profile, data-residency posture, and — critically — the way each model coordinates with the other six systems in the pipeline differ by an order of magnitude. That's where projects die.
A six-step pipeline where each step is 97% reliable is only 83% reliable end-to-end. Most financial institutions discover this after they've already shipped a pilot, presented green metrics to the risk committee, and watched the production system quietly leak errors at the handoffs. The individual models were fine. The coordination between them was never designed. This compounding-error dynamic is well documented in reliability engineering — see the broader analysis at arXiv and the practitioner discussion in O'Reilly's production-AI literature.
Coined Framework
The AI Coordination Gap
The AI Coordination Gap is the compounding reliability and compliance loss that occurs at the handoffs between AI models, retrieval systems, and downstream business systems — not inside any single model. It names why 'better model' decisions rarely move the needle in regulated production environments.
McKinsey's five hallmarks — a clear strategy, a reimagined technology backbone, talent, an operating model, and risk controls — are all, at their core, coordination problems. Not one of them is solved by picking a bigger model. Financial services leaders who understand this stop asking 'GPT-4o or a custom SLM?' and start asking 'where does reliability leak between my components, and which model minimizes that leak at each seam?'
This article breaks the decision into a six-layer framework. For each layer, I'll show what actually happens in production, what it costs, and which deployment pattern — custom SLM, off-the-shelf LLM, or hybrid — wins. Then we'll look at real named deployments at JPMorgan, Morgan Stanley, and Klarna, and close with an operator FAQ covering agentic AI, multi-agent orchestration, RAG vs fine-tuning, LangGraph, MCP, and the biggest failures to learn from.
83%
End-to-end reliability of a 6-step pipeline where each step is 97% reliable
[Compounding error analysis, arXiv 2025](https://arxiv.org/)
$1T
Projected annual value of generative AI for global banking
[McKinsey, 2025](https://www.mckinsey.com/)
10-30x
Inference cost reduction of a fine-tuned SLM vs frontier LLM at equivalent task accuracy
[NVIDIA SLM economics study, 2025](https://research.nvidia.com/)
In regulated finance, the winning AI teams are not the ones with the best model. They're the ones who solved the handoffs no one else designed.
What Is a Custom SLM, and When Does It Beat an Off-the-Shelf LLM?
A Small Language Model (SLM) is a language model — typically 1B to 13B parameters — narrow enough to run on a single GPU or even on-premise CPU inference, and fine-tuned on your domain data. In financial services, that domain data is gold: decades of adjudicated disputes, KYC decisions, credit memos, and regulatory correspondence. You own it. You've paid for it in time and compliance overhead. An SLM lets you actually use it.
An off-the-shelf LLM — GPT-4o, Claude 3.7 Sonnet, Gemini 2.5 — is a frontier general model accessed via API. Brilliant at reasoning across novel situations. Also expensive per token, harder to pin to a data-residency requirement, and a moving target you don't control. That last part matters more than most teams admit until a silent model update breaks their evals at 2am. The open-weight alternative — Meta's Llama family and similar releases documented on Hugging Face — is what makes self-hosted SLMs viable in the first place.
A fine-tuned 7B SLM handling 90% of your standardized dispute-triage volume, with an off-the-shelf LLM handling the 10% edge cases, routinely cuts inference cost by 60-80% versus routing everything to GPT-4o — while improving accuracy on the common cases.
The decision variables that actually matter
Forget benchmark leaderboards. In a regulated bank, the deploy decision turns on five variables:
Task variance: Low-variance, high-volume tasks (dispute triage, statement classification, KYC document extraction) favor SLMs. High-variance reasoning tasks (novel fraud narratives, complex client advisory) favor LLMs.
Data residency: If the workload can't leave your VPC or region, an open-weight SLM you host yourself often wins by default — not because it's smarter, but because it's the only option your legal team will sign off on.
Latency budget: Sub-200ms interactive experiences favor a co-located SLM over a network round-trip to a frontier API.
Auditability: Frozen SLM weights give you a reproducible, versioned model for model-risk-management (MRM) sign-off. A vendor LLM that silently updates does not. I've watched MRM reviews stall for months over this exact issue.
Volume economics: Below roughly 1M tokens/day, API LLMs are cheaper all-in. Cross that threshold and self-hosted SLMs start generating serious savings.
DimensionCustom Fine-Tuned SLMOff-the-Shelf LLM (API)Hybrid (Router)
Per-task cost at scaleLowest (10-30x cheaper)HighestLow
Accuracy on novel reasoningModerateHighestHighest
Data residency controlFull (self-hosted)LimitedConfigurable
MRM / audit reproducibilityStrong (frozen weights)Weak (vendor updates)Moderate
Time to first deployment8-16 weeks1-3 weeks10-18 weeks
Latency (interactive)<200ms co-located300-900msVariable
Ongoing ops burdenHighLowHigh
The deployment decision is not binary. Most winning financial-services architectures are hybrid routers that send low-variance volume to an SLM and edge cases to a frontier LLM — closing the AI Coordination Gap by design.
The Six Layers of the AI Coordination Gap Framework
The AI Coordination Gap framework decomposes any financial-services AI system into six coordination layers. The model choice — SLM vs LLM — is only one of them. The other five are where projects actually live or die.
Coined Framework
The AI Coordination Gap
The AI Coordination Gap treats reliability as a property of the seams between components, not the components themselves. In a bank, the gap is where a 97%-accurate extraction model feeds a 96%-accurate classifier feeds a 98%-accurate policy check — and the compounded output silently fails a compliance audit.
Layer 1 — The Retrieval Layer (RAG)
Before any model reasons, it needs the right context. In finance, that means retrieving the customer's actual transaction history, the current version of a regulation, or the exact terms of a product. This is Retrieval-Augmented Generation (RAG), typically built on a vector database like Pinecone or an open alternative.
The coordination failure here is subtle and I've seen it burn teams repeatedly: retrieval that returns 'plausible but stale' documents. A model grounded on last quarter's fee schedule will confidently produce a compliant-sounding, wrong answer. Recency filtering and source-version metadata are not optional in regulated environments. The docs on most vector databases don't emphasize this enough. Our full RAG design guide covers the version-metadata pattern in depth.
Layer 2 — The Reasoning Layer (SLM or LLM)
This is the layer everyone fixates on — and usually the least broken one. Given retrieved context, does the reasoning happen on a fine-tuned SLM or a frontier LLM? For a well-scoped, high-volume task, an SLM fine-tuned on 5,000-50,000 labeled examples typically matches an LLM within a few points at a fraction of the cost. Learn more about how these fit into broader multi-agent systems.
Layer 3 — The Orchestration Layer (LangGraph / AutoGen / CrewAI)
Orchestration decides which model runs when, how state passes between steps, and what happens on failure. LangGraph (production-ready, graph-based state machines), AutoGen (Microsoft, conversation-first), and CrewAI (role-based, faster to prototype but I wouldn't trust it in a regulated flow without significant hardening) are the leading frameworks. This layer is where the router lives that sends volume to your SLM and edge cases to your LLM.
Layer 4 — The Tool/Action Layer (MCP)
Reasoning is useless if it can't act — pulling a live balance, filing a dispute, flagging a transaction. The Model Context Protocol (MCP), introduced by Anthropic, is fast becoming the standard interface between models and enterprise systems. It turns brittle, bespoke integrations into a governed, discoverable tool layer. Explore how this connects to orchestration patterns.
Layer 5 — The Guardrail & Compliance Layer
This is the layer that separates a demo from a regulated deployment. It validates outputs against policy, redacts PII, checks for prohibited advice, and produces an audit trail. Every model output that touches a customer needs a deterministic gate behind it — full stop. This layer is non-negotiable, and it's where most vendor pilots quietly cut corners to hit their demo deadlines. Regulatory guidance like the NIST AI Risk Management Framework increasingly expects exactly this kind of deterministic control.
Layer 6 — The Observability & Evaluation Layer
You can't manage the AI Coordination Gap if you can't see it. This layer traces every request across all five prior layers, measures per-seam reliability, and feeds an evaluation harness. Without it you're flying blind on the exact metric that predicts production failure. See our guide to enterprise AI observability.
Hybrid SLM/LLM Dispute-Triage Pipeline in a Regulated Bank
1
**RAG Retrieval (Pinecone + version metadata)**
Pulls the customer's transaction history and the current fee schedule. Recency filter rejects stale document versions. Latency ~80ms.
↓
2
**LangGraph Router**
Classifies task variance. Routes 90% standardized disputes to the SLM; 10% novel narratives to GPT-4o. Decision logged for audit.
↓
3
**Fine-Tuned 7B SLM (self-hosted, VPC)**
Adjudicates standard disputes. Frozen weights, versioned for MRM. Inference ~150ms, ~20x cheaper than API LLM.
↓
4
**MCP Tool Layer**
Executes governed actions: file provisional credit, flag for review. Every tool call permissioned and logged.
↓
5
**Guardrail Gate (deterministic)**
Validates against dispute-resolution policy, redacts PII, blocks prohibited language. Fails closed on violation.
↓
6
**Observability (per-seam tracing)**
Traces reliability at every handoff, feeds evaluation harness, surfaces the exact seam where reliability leaks.
This sequence matters because reliability compounds down the chain — the guardrail and observability layers exist specifically to catch the AI Coordination Gap before it reaches a customer.
Stop asking 'which model is smartest.' Start asking 'where does reliability leak between my components' — that question chooses the model for you.
What Most Financial Services Companies Get Wrong About AI Technology
After watching dozens of banking AI technology programs crater, the same mistakes keep showing up. They're not model mistakes. They're coordination mistakes.
❌
Mistake: Benchmarking models in isolation
Teams run GPT-4o vs a custom SLM on a static eval set, pick the winner by 2 points, and ship it. The eval never measured the compounded pipeline, so the 83% end-to-end reliability problem is invisible until production.
✅
Fix: Build a LangGraph-based end-to-end eval harness that measures reliability at every seam, not just the model. Optimize the weakest handoff first.
❌
Mistake: Treating RAG recency as optional
Vector search returns semantically similar but out-of-date regulatory documents. The model produces a confident, non-compliant answer grounded on last year's rule.
✅
Fix: Attach version and effective-date metadata to every chunk in Pinecone and filter on recency at query time. Never trust semantic similarity alone in regulated retrieval.
❌
Mistake: Fine-tuning when you should be retrieving
Teams fine-tune an SLM to 'know' policy facts that change quarterly. When the policy updates, the model is silently wrong and needs re-training. I've seen this cost a team two months of rework they didn't budget for.
✅
Fix: Fine-tune for behavior and format, use RAG for facts that change. Facts belong in retrieval; skills belong in weights.
❌
Mistake: No deterministic guardrail behind the model
Relying on the model to 'not say the wrong thing.' In regulated finance, probabilistic guardrails are not a control the risk committee will accept. This will fail your MRM review.
✅
Fix: Place a deterministic, rules-based guardrail gate after every generative step that fails closed. The model proposes; deterministic code disposes.
The single highest-ROI move in most banking AI programs isn't upgrading the model — it's adding per-seam observability. One institution found 71% of its errors originated in a single retrieval-to-reasoning handoff that no one was measuring.
How to Implement: A Practical SLM/LLM Hybrid in Financial Services
Here's the implementation path that consistently works. Start with an off-the-shelf LLM to prove the workflow, instrument everything, then migrate high-volume paths to a fine-tuned SLM once you have labeled production data. Don't skip steps — I've watched teams try to jump straight to fine-tuning and spend three months building a model they later scrapped because they'd labeled the wrong thing.
Andrew Ng, founder of DeepLearning.AI, has repeatedly argued that the fastest teams prototype with the strongest available model and optimize cost later — a pattern that maps directly onto this migration path. Anthropic's applied research lead Erik Schluntz has emphasized that MCP-based tool governance is what makes agentic workflows safe to run in regulated settings. And McKinsey senior partner Alexander Sukharevsky notes that the banks pulling ahead are those that rebuilt their technology backbone — the orchestration and data plumbing — not just their model access. All three are describing the same thing: coordination first, model optimization second.
Step 1 — Prototype the full pipeline on an off-the-shelf LLM
Wire up RAG, orchestration, tools, and guardrails end-to-end using GPT-4o or Claude 3.7. Get to a working demo in 1-3 weeks. Resist the urge to fine-tune anything yet — you don't have production labels, and you won't know which task variants actually show up at volume until you've shipped something real.
Step 2 — Instrument every seam
Add tracing at each of the six layers. Measure reliability per handoff. This is where you'll find your real bottleneck, which is almost never the model. Every time I've done this exercise, the retrieval-to-reasoning handoff is the worst offender — and nobody was looking at it.
Python — LangGraph router with SLM/LLM cost split
Route low-variance tasks to a fine-tuned SLM, edge cases to a frontier LLM
from langgraph.graph import StateGraph, END
def classify_variance(state):
# deterministic classifier scores task complexity 0-1
score = state['complexity_score']
return 'slm' if score < 0.6 else 'llm' # 90% route to SLM
def slm_node(state):
# self-hosted 7B fine-tuned model, ~20x cheaper per call
state['answer'] = slm_client.generate(state['prompt'])
return state
def llm_node(state):
# frontier LLM for novel reasoning only
state['answer'] = llm_client.generate(state['prompt'])
return state
graph = StateGraph(dict)
graph.add_node('slm', slm_node)
graph.add_node('llm', llm_node)
graph.add_conditional_edges('router', classify_variance,
{'slm': 'slm', 'llm': 'llm'})
graph.add_edge('slm', 'guardrail') # every path hits deterministic gate
graph.add_edge('llm', 'guardrail')
app = graph.compile()
Step 3 — Harvest labels and fine-tune your SLM
Once the LLM pipeline has processed 5,000-50,000 real cases (with human review on a sample), you have a labeled dataset. Fine-tune an open-weight 7B model on it. Freeze the weights and version them for MRM. You can accelerate this stage with pre-built components — explore our AI agent library for reference router and guardrail implementations.
Step 4 — Migrate high-volume paths and measure savings
Flip the router so standardized volume flows to the SLM. Watch cost drop 60-80% while accuracy on common cases holds or improves. Keep the LLM for the long tail. For teams standardizing this pattern, our workflow automation and AI agents guides cover the deployment templates in depth. Non-technical teams can prototype the orchestration in n8n before hardening in LangGraph, and can start from a vetted template in our production-ready agent templates.
The production router in practice: 90% of dispute volume flows to a self-hosted fine-tuned SLM, with only novel cases escalating to a frontier LLM — every path passing through the same deterministic guardrail gate.
[
▶
Watch on YouTube
Small Language Models in Production: Fine-Tuning and Cost Economics
Enterprise AI • SLM deployment patterns
](https://www.youtube.com/results?search_query=small+language+models+enterprise+fine+tuning+production)
Real Deployments: JPMorgan, Morgan Stanley, and Klarna
Theory is cheap. Here's what named financial and fintech firms actually shipped.
Coined Framework
The AI Coordination Gap
In each deployment below, the differentiator was not model quality — it was how well the firm engineered the seams between retrieval, reasoning, tools, and compliance. The winners closed the AI Coordination Gap; the cautionary tales did not.
Morgan Stanley — LLM + retrieval for advisor assist
Morgan Stanley's advisor-facing assistant, built with OpenAI, retrieves from its curated research library and lets 16,000+ advisors query decades of internal knowledge. The critical design choice was a heavily governed RAG layer over vetted internal documents — not open-ended generation. The coordination work — grounding, source attribution, guardrails — was the actual product. The model was just the reasoning engine underneath it.
JPMorgan — orchestrated, in-house tooling
JPMorgan's internal LLM Suite gives tens of thousands of employees access to models behind a governed orchestration and data-control layer that keeps sensitive data inside the firm's boundary. It's a textbook example of prioritizing the technology backbone and risk controls — two of McKinsey's five hallmarks — over chasing the single best model. They built the plumbing first. The model was almost secondary.
Klarna — the cautionary counterweight
Klarna's AI customer-service agent famously handled the work of hundreds of agents. Then in 2025 the company publicly recalibrated, rehiring human agents for complex cases after discovering that pure automation degraded experience on the long tail, as widely reported and covered in depth by Bloomberg. The lesson is precisely the hybrid thesis: route the standardized 90% to automation, keep humans and frontier reasoning for the edge cases. Klarna learned the AI Coordination Gap the expensive way — in public.
Klarna didn't fail at AI. It failed at knowing which 10% of cases should never have been fully automated. That's the whole game in regulated services.
16,000+
Morgan Stanley advisors using its OpenAI-powered retrieval assistant
[OpenAI / Morgan Stanley, 2024](https://openai.com/index/morgan-stanley/)
60-80%
Typical inference-cost reduction from migrating high-volume paths to a fine-tuned SLM
[SLM economics analysis, arXiv 2025](https://arxiv.org/)
5
Hallmarks of effective banking AI strategy (all coordination-related)
[McKinsey, 2026](https://www.mckinsey.com/)
What Comes Next: Predictions for Regulated AI Through 2027
2026 H2
**MCP becomes the default enterprise tool interface**
With Anthropic's Model Context Protocol gaining rapid adoption and vendor support, banks standardize their tool/action layer on MCP, retiring bespoke integrations. This directly narrows the AI Coordination Gap at Layer 4.
2027 H1
**Fine-tuned SLMs handle the majority of banking AI volume**
As NVIDIA's SLM economics research predicts, cost pressure and data-residency rules push the bulk of high-volume, low-variance workloads onto self-hosted SLMs, with frontier LLMs reserved for edge cases.
2027 H2
**Per-seam reliability becomes a regulated metric**
Model-risk-management frameworks evolve to require end-to-end pipeline reliability evidence, not just model-level evals — formalizing the AI Coordination Gap as a compliance concept. If you're not measuring this already, you'll be required to.
The trajectory for regulated AI: tool standardization via MCP, SLM dominance for volume, and per-seam reliability entering formal model-risk-management — each milestone a step toward closing the AI Coordination Gap.
Frequently Asked Questions
What is agentic AI?
Agentic AI refers to systems where a language model doesn't just generate text but plans, uses tools, and takes multi-step actions toward a goal — retrieving data, calling APIs, and making decisions along the way. In financial services, an agentic system might triage a dispute, pull the transaction record, apply policy, and file provisional credit. Frameworks like LangGraph (production-ready) and AutoGen (Microsoft) manage this loop. The key distinction from a chatbot is autonomy over actions. In regulated settings, every agent action must pass through a deterministic guardrail and be logged for audit — otherwise you inherit the compounding reliability problem the AI Coordination Gap describes. Start scoped: one task, tightly governed tools, full observability.
How does multi-agent orchestration work?
Multi-agent orchestration coordinates several specialized agents — each handling a sub-task — under a controller that routes work and manages shared state. A retrieval agent fetches context, a reasoning agent (SLM or LLM) decides, a tool agent acts, and a guardrail agent validates. Orchestration frameworks like LangGraph model this as a state graph with explicit edges and failure handling; CrewAI uses role-based teams; AutoGen uses conversation. The critical engineering challenge is the handoff — state passed between agents is exactly where reliability compounds and leaks. Well-designed orchestration measures reliability at every seam. Poorly designed orchestration produces a demo that works 95% of the time and fails silently the rest, which is unacceptable in banking. Instrument each edge before scaling.
What companies are using AI agents?
In financial services, Morgan Stanley deploys an OpenAI-powered retrieval assistant to 16,000+ advisors, JPMorgan runs an internal LLM Suite behind a governed data boundary, and Klarna built a large-scale customer-service agent (later recalibrated to a human-plus-AI hybrid). Beyond finance, Anthropic, Google DeepMind, and countless enterprises run agentic systems in production. The pattern across successful enterprise AI deployments is consistent: they invest heavily in the coordination layers — retrieval grounding, tool governance via MCP, and compliance guardrails — rather than chasing the largest model. The cautionary cases, like Klarna's over-automation, teach the same lesson from the other direction: know which fraction of your volume genuinely needs human or frontier-model handling.
What is the difference between RAG and fine-tuning?
RAG (Retrieval-Augmented Generation) injects external facts into a model's context at query time by retrieving relevant documents from a vector database like Pinecone. Fine-tuning adjusts the model's weights on your data to change its behavior, tone, or task skill. The rule: use RAG for facts that change (policies, balances, regulations) and fine-tuning for skills and format that stay stable (how to structure a dispute adjudication). A common, costly mistake is fine-tuning a model to memorize facts that update quarterly — it goes silently stale. In a banking hybrid, you'll typically use both: fine-tune a compact SLM for consistent task behavior, and layer RAG on top for current, auditable facts. See our deeper RAG breakdown for retrieval design patterns.
How do I get started with LangGraph?
Start with the LangGraph documentation and install via pip install langgraph. Model your workflow as a state graph: define a shared state object, add nodes for each step (retrieval, reasoning, tool call, guardrail), and connect them with conditional edges that route based on state. Begin with a single linear graph before adding branching. The framework is production-ready and gives you checkpointing, human-in-the-loop pauses, and failure handling out of the box — all essential in regulated finance. A practical first project: a two-node graph that routes low-complexity tasks to a cheap model and complex ones to a frontier LLM, then passes both through a deterministic guardrail. Our LangGraph guide walks through a complete banking router example.
What are the biggest AI failures to learn from?
The most instructive failures share a root cause: over-trusting a single model and under-engineering the coordination around it. Klarna's over-automation of customer service — later recalibrated with rehired human agents — showed the cost of ignoring the long-tail edge cases. Broadly, deployments fail when teams benchmark models in isolation, skip recency filtering in RAG (producing confident, non-compliant answers on stale data), and rely on probabilistic rather than deterministic guardrails. The compounding-error math is unforgiving: a six-step pipeline at 97% per step is only 83% reliable end-to-end. Every one of these is a coordination failure, not a model failure. The fix is per-seam observability, deterministic gates behind every generative step, and a hybrid architecture that reserves humans and frontier models for the cases that genuinely need them.
What is MCP in AI?
MCP (Model Context Protocol) is an open standard introduced by Anthropic that defines how AI models connect to external tools, data sources, and systems in a governed, discoverable way. Instead of writing brittle custom integrations for every data source, you expose them through a standardized MCP interface that the model can discover and call with proper permissioning and logging. In financial services this matters enormously: it turns the tool/action layer — Layer 4 of the AI Coordination Gap — from a security liability into a controlled, auditable surface. MCP adoption is accelerating across enterprise vendors through 2026, and it's becoming the default way to give agents safe access to banking systems. If you're building agentic workflows today, standardize your tool layer on MCP rather than one-off API wrappers.
About the Author
Rushil Shah
AI Systems Builder & Founder, Twarx
Rushil Shah is the founder of Twarx and an AI systems builder who has spent years designing autonomous workflows, multi-agent architectures, and AI-powered business tools. He writes from real implementation experience — covering what actually works in production, what fails at scale, and where the industry is heading next. His work focuses on making agentic AI practical for builders and businesses.
LinkedIn · Full Profile
This article was originally published on Twarx. Follow for daily deep dives on AI agents and automation.
Top comments (0)