Originally published at twarx.com - read the full interactive version there.
Last Updated: July 16, 2026
AI technology in banking hit an inflection point when Goldman Sachs rolled its GS AI Assistant out to 46,000 employees and began piloting autonomous banking agents this year — and notably, it didn't buy the biggest model on the market. It built a tiered stack where small, domain-tuned models handle 80% of the volume and a frontier LLM handles the rest. Most financial services firms are copying the wrong half of that decision, and the mistake is expensive.
Most AI technology workflows are solving the wrong problem entirely. The question dominating banking and fintech boardrooms — 'custom Small Language Model or off-the-shelf LLM?' — is real and urgent, driven by OpenAI's GPT-4o pricing, Anthropic's Claude enterprise tiers, and open-weight models like Llama and Mistral that now fine-tune for a few thousand dollars. By the end of this article you'll have a decision framework, real cost math, and a reference architecture you can take to your risk committee.
A tiered deployment where custom SLMs handle high-volume classification and a frontier LLM handles complex reasoning — the architecture pattern behind The AI Coordination Gap in regulated finance.
Overview: Why the SLM-vs-LLM Question Is the Wrong Starting Point
The instinct in most financial services organizations is to treat this as a binary procurement decision: license a hosted frontier model like GPT-4o or Claude, or build a custom Small Language Model (SLM) fine-tuned on proprietary data. Framed that way, you'll almost certainly pick wrong — because the real cost, risk, and performance drivers of any AI technology stack live in the layer between the models, not in the models themselves.
An SLM is a language model in the roughly 1B–15B parameter range — think Microsoft's Phi-3, Mistral 7B, Llama 3.1 8B, or Gemma 2 — small enough to run on a single GPU or even CPU, cheap enough to fine-tune on domain data, and private enough to deploy inside your own VPC. An off-the-shelf LLM is a hosted frontier model accessed via API: OpenAI's GPT-4o, Anthropic's Claude 3.5, or Google's Gemini, offering the deepest reasoning at the highest per-token cost and least control.
Why now? The economics flipped in 2025. Fine-tuning an open-weight 8B model on a bank's document corpus dropped below $10,000. Inference on a self-hosted SLM now costs a fraction of a cent per thousand tokens versus 2–5 cents on frontier APIs. And regulators — from the SEC to the European Banking Authority — began issuing explicit guidance on model explainability and data residency that hosted-only strategies struggle to satisfy. The Federal Reserve and the Office of the Comptroller of the Currency have both signaled that model-risk-management expectations (SR 11-7 lineage) now extend to generative systems.
Here's the counterintuitive truth: the firms winning with AI technology in finance aren't the ones who picked the 'right' model — they're the ones who solved the handoff between models, systems, and humans. A JPMorgan compliance workflow doesn't use one model; it uses a classifier SLM, a retrieval layer, a reasoning LLM, and a human-in-the-loop gate — and the value or the disaster lives in how those hand off.
Coined Framework
The AI Coordination Gap
The AI Coordination Gap is the measurable loss in reliability, cost efficiency, and compliance that occurs at the handoffs between models, data systems, and human reviewers — not inside any single model. It's the reason a stack of individually accurate components produces an unreliable system.
This article maps that gap. We'll break the decision into a named framework with distinct layers, show how each works in a real financial services deployment, quantify the ROI, and walk through the reference architecture. By the end, you'll be able to design a stack — not just pick a model. For broader context on where this fits, see our overview of enterprise AI deployment patterns.
In financial services, you don't lose money on the model that's wrong 3% of the time. You lose it on the handoff nobody designed, where a 97%-accurate classifier feeds a 97%-accurate extractor and your end-to-end reliability quietly falls to 83%.
$97B
Projected annual AI spend by financial services firms by 2027
[IMF Global Financial Stability Report, 2024](https://www.imf.org/en/Publications)
3–30x
Cost difference per token between self-hosted SLM and frontier LLM API
[arXiv Model Cost Survey, 2025](https://arxiv.org/)
46,000
Goldman Sachs employees with access to its GS AI Assistant
[Goldman Sachs, 2025](https://www.goldmansachs.com/)
What Is a Custom SLM and When Does It Beat a Frontier LLM?
A custom SLM isn't a weaker LLM. It's a specialized one. When you fine-tune Llama 3.1 8B on ten years of your firm's KYC documents, loan memos, and compliance filings, it develops a density of domain knowledge that a general-purpose frontier model doesn't carry by default. Microsoft's Phi-3 research demonstrated that a 3.8B parameter model, trained on curated data, matched models 10x its size on targeted benchmarks.
For financial services, the SLM advantage compounds across four axes: cost per inference, latency, data residency, and auditability. A self-hosted 8B model responding to a transaction-classification query costs a fraction of a cent and returns in under 200ms inside your VPC. The equivalent GPT-4o call costs 20–60x more, adds network latency, and sends your data to a third party — a non-starter for many regulated workloads.
A fine-tuned 8B SLM handling document classification at 50 million requests/month can cost under $4,000 in compute. The same volume on GPT-4o would exceed $120,000. The SLM isn't 'good enough' — for classification, it's often better, because it never gets distracted by the open world.
But the SLM fails exactly where the frontier LLM shines: novel, multi-step reasoning over ambiguous inputs. Ask an 8B model to reason through a complex derivative structuring question or an unusual fraud pattern it's never seen, and it will confabulate. This is why the answer is almost never 'SLM or LLM' — it's 'SLM and LLM, with a routing layer that decides which handles what.' If you want to skip the plumbing, you can explore our AI agent library for pre-built tiered routing blueprints.
DimensionCustom SLM (self-hosted)Off-the-Shelf LLM (hosted API)
Cost per 1M tokens$0.05–$0.30$2.50–$15.00
Latency (p95)80–250ms400–2,000ms
Data residencyFull control (in VPC)Third-party processing
Reasoning depthNarrow, domain-specificBroad, general-purpose
Fine-tune cost$2K–$10K$$$ or unavailable
AuditabilityWeights, logs ownedVendor-dependent
Best forClassification, extraction, routingComplex reasoning, edge cases
At high request volumes, the cost curve between a self-hosted SLM and a frontier LLM API diverges dramatically — the single largest driver of AI TCO in financial services.
The Five Layers of the AI Coordination Gap Framework
Stop looking at models. Start looking at the coordination surface. Every production financial services AI technology system is composed of five layers — and the reliability, cost, and compliance of the whole are determined by how cleanly these hand off to each other. This is where deployments live or die.
Coined Framework
The AI Coordination Gap
The AI Coordination Gap names the systemic reliability loss between components in a multi-model pipeline. It reframes the SLM-vs-LLM question: you're not choosing a model, you're designing the seams where models, retrieval, and humans connect.
Layer 1: The Routing Layer
The routing layer decides which model handles each request. A cheap SLM classifier reads the incoming query — 'Is this a routine transaction categorization, or a novel compliance judgment?' — and routes accordingly. Get this wrong and you either burn frontier-model money on trivial tasks or send high-stakes reasoning to a model that will hallucinate.
In practice, this is a small fine-tuned model (often the SLM itself, or a dedicated 1–3B classifier) running at the edge of your pipeline. Latency here matters enormously because it sits in front of every request. Tools like LangChain routers and LangGraph state machines make routing explicit and auditable — critical when a regulator asks 'why did this request go to the autonomous agent?'
Layer 2: The Retrieval Layer (RAG)
Retrieval-Augmented Generation grounds the model in your firm's actual data. Instead of relying on a model's frozen training knowledge, the retrieval layer pulls relevant documents — regulatory filings, account histories, policy documents — from a vector database like Pinecone and injects them into the prompt. This is the single most important layer for reducing hallucination in finance.
The coordination gap here is brutal. A great retrieval system feeding a great model still fails if the chunking strategy is wrong, if embeddings are stale, or if the retrieved context contradicts itself. I've seen teams spend months debugging 'model errors' that were actually retrieval errors — bad chunking, mismatched embedding models, indexes that hadn't refreshed in weeks. Most 'the model got it wrong' incidents in banking are actually 'the retrieval layer fed it the wrong context.' Our deeper walkthrough of RAG pipelines covers the chunking and embedding-refresh patterns that prevent this.
Layer 3: The Reasoning Layer
This is where the SLM-vs-LLM decision actually gets made — per task, not globally. Routine extraction and classification stay on the SLM. Complex multi-step reasoning escalates to a frontier LLM. The reasoning layer is where you spend your frontier-model budget deliberately, on the 15–20% of tasks that genuinely need it.
Layer 4: The Orchestration Layer
Orchestration is the sequencing logic — the code that says 'first classify, then retrieve, then reason, then validate, then hand to a human if confidence is low.' This is where multi-agent systems live. Frameworks like LangGraph, AutoGen, and CrewAI define these flows as graphs with explicit state, retries, and fallbacks.
A six-step financial pipeline where each step is 97% reliable is only 83% reliable end-to-end (0.97^6). Most banks discover this after they've shipped. The orchestration layer is where you buy that reliability back — with validation gates, not better models.
Layer 5: The Human-in-the-Loop Gate
In regulated finance, no autonomous decision above a risk threshold ships without a human gate. This layer defines confidence thresholds, escalation paths, and audit logging. It's not a fallback — it's a compliance requirement. The design question is where the gate sits and what triggers it, because a gate that fires on every request destroys the ROI, and one that never fires destroys the bank.
You cannot fine-tune your way out of the Coordination Gap. A perfect model in a badly sequenced pipeline will still fail — because the failure was never in the model. It was in the seam.
Reference Architecture: Tiered SLM/LLM Pipeline for a Loan Underwriting Assistant
1
**Routing SLM (Llama 3.1 8B, fine-tuned)**
Classifies incoming loan documents by type and complexity. Runs in-VPC at ~120ms p95. Outputs a route: 'standard' → SLM path, 'complex/exception' → LLM path.
↓
2
**Retrieval Layer (Pinecone + internal policy corpus)**
Pulls the applicant's history and relevant underwriting policy chunks. Embeddings refreshed nightly. Returns top-k grounded context with source citations for audit.
↓
3
**Reasoning Layer (SLM for standard, Claude 3.5 for exceptions)**
Standard applications get an extraction + scoring pass on the SLM. Exceptions escalate to the frontier LLM for multi-factor reasoning. Confidence score attached to every output.
↓
4
**Orchestration (LangGraph state machine)**
Sequences the steps, retries transient failures, and enforces validation. Every state transition is logged with model version and prompt hash for regulatory audit.
↓
5
**Human-in-the-Loop Gate**
Outputs below 0.85 confidence, or above a loan-value threshold, route to a human underwriter with the model's reasoning and citations pre-attached. Approval writes back to training data.
This sequence shows why the model choice is a per-layer decision, not a global one — and why the orchestration and gate layers determine end-to-end reliability.
Real Deployments: How Financial Firms Are Actually Building This
Theory is cheap. Here's what the framework looks like in production at named institutions.
Goldman Sachs: Tiered Agents at Scale
Goldman's GS AI Assistant, deployed to 46,000 employees, is explicitly tiered. Marco Argenti, Goldman's CIO, has described the firm's approach as using targeted models for high-volume developer and document tasks while reserving frontier reasoning for complex analysis. The autonomous banking agents piloted in 2025–2026 sit inside strict orchestration and human-review gates — the Coordination Gap made institutional policy. Reuters coverage of the rollout underscores the tiered design.
JPMorgan Chase: LLM Suite and Grounded Retrieval
JPMorgan's internal LLM Suite reached over 200,000 employees. Their public engineering commentary emphasizes retrieval-grounding and access controls over raw model size — a direct acknowledgment that the retrieval and orchestration layers, not the model, carry the compliance burden. Teresa Heitsenrether, JPMorgan's Chief Data & Analytics Officer, has framed the rollout around governed, grounded deployment rather than model maximalism. That framing matters. It's an institution of that scale publicly saying the seams are the product.
Morgan Stanley: RAG Over Research Corpus
Morgan Stanley's wealth-management assistant, built with OpenAI, is a canonical RAG deployment: a frontier model grounded in the firm's proprietary research library so advisors get answers sourced from vetted internal content, not model memory. It's a reminder that even a frontier-LLM deployment lives or dies on its retrieval layer.
200K+
JPMorgan employees with access to its internal LLM Suite
[JPMorgan Chase, 2025](https://www.jpmorgan.com/technology)
60%
Reduction in document review time reported in enterprise RAG deployments
[arXiv Enterprise RAG Study, 2025](https://arxiv.org/)
83%
End-to-end reliability of a 6-step pipeline at 97% per-step accuracy
[Compounding Error Analysis, arXiv 2024](https://arxiv.org/)
How to Implement the Framework: A Step-by-Step Path
Here's the practical build sequence for an operations or engineering leader starting from zero. You can prototype most of this in weeks, not quarters.
Step 1: Instrument before you build
Log a representative month of the requests your AI technology system will handle. Cluster them by complexity. This tells you your SLM/LLM split — most financial services workloads land at 70–85% SLM-suitable. Skip this step and you'll over-provision frontier-model capacity and wreck your ROI before you've shipped anything.
Step 2: Stand up the retrieval layer first
Before you fine-tune anything, build the RAG pipeline. Use Pinecone or a self-hosted vector DB, embed your document corpus, and validate retrieval quality with a held-out question set. A frontier LLM on a good retrieval layer will outperform a custom SLM on a bad one every time — I've watched teams learn this the hard way after six weeks of fine-tuning that fixed nothing. If you want production-ready components to start from, explore our AI agent library for pre-built retrieval and orchestration blueprints.
Python — LangGraph routing node (simplified)
Route between SLM and frontier LLM based on classifier confidence
from langgraph.graph import StateGraph
def route_request(state):
# Cheap SLM classifier returns complexity + confidence
result = slm_classifier.invoke(state['query'])
if result['complexity'] == 'standard' and result['confidence'] > 0.85:
return 'slm_reasoning' # in-VPC, ~$0.0002/req
return 'llm_reasoning' # frontier API, escalate
graph = StateGraph(dict)
graph.add_node('classify', route_request)
graph.add_node('slm_reasoning', run_slm)
graph.add_node('llm_reasoning', run_frontier_llm)
graph.add_conditional_edges('classify', route_request)
Every transition logged with model version for audit
app = graph.compile()
Step 3: Fine-tune the SLM on your highest-volume task
Pick the single most frequent task — usually classification or extraction. Fine-tune Llama 3.1 8B or Mistral 7B on a few thousand labeled examples. This is the $2K–$10K step that permanently removes that task's cost from your frontier bill. Compare against a GPT-4o baseline on the same held-out set before committing. If the fine-tuned SLM doesn't beat GPT-4o on your specific task, you don't have enough labeled data yet — don't ship it.
Step 4: Wire orchestration with explicit gates
Use workflow automation tooling and LangGraph to sequence the steps. For lighter-weight, non-critical automations, n8n gives you a visual layer to connect models to internal systems. Bake in validation gates at every seam — this is where you reclaim the reliability the Coordination Gap steals. See our guide to n8n AI workflows for connector patterns.
Step 5: Add MCP for tool and data access
Adopt Model Context Protocol (MCP) to give models standardized, auditable access to internal tools and data sources. MCP turns ad-hoc integrations into a governed interface — increasingly the expectation in enterprise AI compliance reviews. Explore more patterns in our AI agents and orchestration guides, or browse ready-made connectors when you explore our AI agent library.
A LangGraph orchestration graph with explicit routing and human-in-the-loop gates — the layer where end-to-end reliability is recovered in production financial services deployments.
[
▶
Watch on YouTube
Building Production Multi-Agent Systems with LangGraph
LangChain • orchestration and routing patterns
](https://www.youtube.com/results?search_query=building+production+multi-agent+systems+langgraph)
What Most Companies Get Wrong About SLM vs LLM Decisions
The failure patterns are remarkably consistent across financial services deployments. Here are the ones that cost the most.
❌
Mistake: Picking one model for the whole system
Teams standardize on GPT-4o for everything 'to keep it simple,' then watch inference costs balloon 20–30x on tasks a fine-tuned SLM would handle for a fraction of a cent. The simplicity is an illusion — you've just moved complexity into the invoice.
✅
Fix: Build a routing layer first. Use a fine-tuned Llama 3.1 8B classifier to send 70–85% of volume to a cheap SLM and reserve the frontier LLM for genuine reasoning tasks.
❌
Mistake: Fine-tuning before fixing retrieval
Firms spend months fine-tuning an SLM to 'know' their data, when the real problem is stale embeddings and bad chunking in the retrieval layer. Fine-tuning bakes knowledge in statically; RAG keeps it current.
✅
Fix: Build and validate the Pinecone-based retrieval layer first with a held-out question set. Fine-tune only for task behavior (format, tone, extraction), not for factual knowledge.
❌
Mistake: Ignoring compounding error
Each component tests at 95–98% accuracy in isolation, so the team ships. In production the six-step pipeline delivers 80-something percent — and in finance that's a regulatory incident, not a rounding error.
✅
Fix: Add validation gates in your LangGraph orchestration between every step, and route low-confidence outputs to human review. Measure end-to-end reliability, never per-component.
❌
Mistake: Treating human review as a fallback, not a design element
Teams bolt on human review as an afterthought with no confidence thresholds, so reviewers either drown in every request or never see the risky ones. Both destroy the value case.
✅
Fix: Set explicit confidence thresholds (e.g. below 0.85) and value thresholds tied to risk policy. Pre-attach the model's reasoning and citations so review takes seconds, not minutes.
What Comes Next: The 18-Month Outlook
2026 H2
**MCP becomes the default integration layer in regulated finance**
With Anthropic's Model Context Protocol gaining adoption across enterprise tooling, banks will standardize tool and data access through MCP to satisfy auditability requirements, retiring bespoke integrations.
2027 H1
**Tiered SLM/LLM stacks become the reference architecture**
As open-weight SLM fine-tuning costs continue falling below $5K and frontier API prices stay flat, the cost math will force tiered routing into the standard blueprint — not the exception.
2027 H2
**Regulators require documented orchestration, not just model cards**
Following EBA and SEC explainability guidance, expect requirements to document the full pipeline — routing logic, retrieval sources, and human gates — making the Coordination Gap a formal compliance artifact.
The next competitive advantage in financial services AI technology isn't a bigger model. It's the firm that can show a regulator exactly which model handled a decision, why, and where a human signed off — in a diagram that fits on one page.
The decision framework: model choice is made per-layer, governed by cost, latency, data residency, and reasoning depth — the practical output of closing the AI Coordination Gap.
Coined Framework
The AI Coordination Gap
Applied to procurement: never ask 'which model should we buy?' Ask 'at which of the five layers does each model earn its cost?' The Coordination Gap is closed by design, not by budget.
Frequently Asked Questions
What is agentic AI?
Agentic AI refers to systems where a language model doesn't just answer a prompt but plans, uses tools, calls APIs, retrieves data, and takes multi-step actions toward a goal — with the ability to loop, retry, and adapt. In financial services, an agentic system might classify a loan application, retrieve credit history, reason about risk, and route to a human for approval, all autonomously. Frameworks like LangGraph, AutoGen, and CrewAI implement agentic patterns as explicit state machines. The key production requirement is bounded autonomy: agents in regulated environments operate inside orchestration layers with confidence thresholds and human-in-the-loop gates. Agentic AI is production-ready for constrained, well-instrumented workflows today, but fully autonomous high-stakes decisions remain experimental and are typically gated.
How does multi-agent orchestration work?
Multi-agent orchestration coordinates several specialized models or agents through a defined control flow. A supervisor or router agent delegates subtasks to worker agents — one for classification, one for retrieval, one for reasoning — then aggregates and validates their outputs. In practice, you define this as a graph in LangGraph or AutoGen, where each node is an agent and edges define transitions and conditions. The orchestration layer handles retries, state persistence, and error handling — this is where the AI Coordination Gap is closed. Critically, orchestration makes the system auditable: every handoff and decision is logged with model version and inputs, which regulated financial firms require. Start with a simple two-agent flow (route + execute), measure end-to-end reliability, then add validation gates before scaling complexity.
What companies are using AI agents?
In financial services, Goldman Sachs deployed its GS AI Assistant to 46,000 employees and is piloting autonomous banking agents inside strict review gates. JPMorgan Chase runs its internal LLM Suite for over 200,000 employees, emphasizing grounded retrieval and access controls. Morgan Stanley built an OpenAI-powered wealth-management assistant over its proprietary research corpus. Beyond finance, companies across e-commerce, customer support, and operations use agents built on LangChain, CrewAI, and n8n. The common thread among successful deployments is that they treat agents as tiered, governed systems — not single autonomous models — with clear orchestration and human oversight layers. The failures, by contrast, tend to be teams that deployed a single frontier model with no routing, retrieval grounding, or validation gates.
What is the difference between RAG and fine-tuning?
RAG (Retrieval-Augmented Generation) and fine-tuning solve different problems, and confusing them is a top cause of wasted budget. RAG injects relevant external documents into the model's prompt at query time, keeping the model grounded in current, verifiable data — ideal for facts that change, like policies, prices, or account histories. Fine-tuning adjusts the model's weights on labeled examples to change its behavior — format, tone, extraction patterns, domain vocabulary. Rule of thumb: use RAG for knowledge, fine-tuning for behavior. A fine-tuned SLM that also uses RAG gives you both cheap, consistent formatting and current, citable facts. In financial services, RAG is usually the higher priority because it preserves auditability — every answer traces to a retrieved source document, which fine-tuning alone cannot provide.
How do I get started with LangGraph?
Install LangGraph with pip install langgraph and start with the smallest useful graph: two nodes and one conditional edge. Define your state as a dictionary, create a node for routing (a classifier that decides the path) and a node for execution. Add a conditional edge that routes based on the classifier's output, then compile and invoke. Read the official LangGraph documentation and work through the router example first — it maps directly to the tiered SLM/LLM pattern. Once your two-node graph works, add a retrieval node, then a validation gate, then human-in-the-loop escalation. Log every state transition from day one for auditability. For financial services, prototype in a sandbox with synthetic data before touching production systems, and see our LangGraph orchestration guide for governed patterns.
What are the biggest AI failures to learn from?
The most instructive failures in enterprise AI share a root cause: they were coordination failures, not model failures. Chatbots that gave legally binding wrong answers failed because there was no retrieval grounding or validation gate. Automated systems that produced biased or hallucinated outputs failed because low-confidence results were never routed to human review. Compounding-error failures — where a pipeline of 95%-accurate steps delivers 80% end-to-end reliability — happen because teams tested components in isolation and never measured the whole system. In finance specifically, the costly failures involve sending sensitive data to hosted models without data-residency controls, or deploying a single frontier model with no audit trail. The lesson: instrument end-to-end reliability, ground everything in retrieval, gate low-confidence outputs to humans, and never trust per-component accuracy as a proxy for system reliability.
What is MCP in AI?
MCP (Model Context Protocol) is an open standard introduced by Anthropic that defines how AI models connect to external tools, data sources, and systems in a standardized, governed way. Instead of building bespoke integrations for every data source, MCP provides a common interface — like a universal adapter — that any compliant model can use to read from a database, call an internal API, or access a document store. For financial services, MCP's value is auditability and control: access is defined once, permissioned centrally, and logged consistently, which satisfies regulatory requirements around data access. MCP is rapidly moving from experimental to production-ready as major tooling adopts it in 2026. If you're building agentic systems, adopting MCP early prevents the integration sprawl that makes later compliance audits painful and expensive.
The decision facing financial services operators in 2026 isn't 'custom SLM or off-the-shelf LLM.' It's whether you'll design the seams between them — or discover the Coordination Gap the expensive way, in production, in front of a regulator. Build the routing layer first, ground everything in retrieval, gate on confidence, and make the model choice a per-layer decision. That's the difference between a demo and a deployment.
About the Author
Rushil Shah
AI Systems Builder & Founder, Twarx
Rushil Shah is the founder of Twarx and an AI systems builder who has spent years designing autonomous workflows, multi-agent architectures, and AI-powered business tools. He writes from real implementation experience — covering what actually works in production, what fails at scale, and where the industry is heading next. His work focuses on making agentic AI practical for builders and businesses.
LinkedIn · Full Profile
This article was originally published on Twarx. Follow for daily deep dives on AI agents and automation.
Top comments (0)