Large language models make it relatively easy to build a chatbot that can answer general questions.
Building one that can safely work with banking information is a different engineering problem.
A banking AI assistant may need to work with internal policies, product documentation, financial reports, compliance material, and enterprise APIs. Much of this information can change over time, and different users may have different access permissions.
That means simply sending a user's question to an LLM is rarely enough.
A more practical architecture combines retrieval, access control, application logic, and LLM generation.
This article walks through a simplified architecture for building such a system using Retrieval-Augmented Generation (RAG).
What Are We Actually Building?
Let's assume we're building an internal AI assistant for bank employees.
An employee might ask:
"What is the current process for handling an international payment exception?"
The system should:
- Understand the question.
- Find relevant internal documents.
- Check whether the employee can access those documents.
- Provide the relevant context to the LLM.
- Generate an answer based on that context.
- Return the supporting sources where appropriate.
The basic flow is:
User
↓
Application
↓
Authentication / Authorization
↓
Query Processing
↓
Document Retrieval
↓
Context Construction
↓
LLM
↓
Response Validation
↓
Answer + Sources
The LLM is therefore only one component of the application.
Why RAG Makes Sense for Banking
LLMs have broad knowledge, but an organization's internal banking policies aren't necessarily part of the model's training data.
Even when a model knows something about banking, that doesn't mean its answer reflects the organization's latest internal procedure.
RAG addresses this by retrieving information from an external knowledge source before generating the answer.
A simplified RAG pipeline looks like:
Question
↓
Embedding / Search
↓
Relevant Documents
↓
Context
↓
LLM
↓
Generated Answer
The knowledge base might contain:
- Internal policies
- Product documentation
- Process manuals
- Support documentation
- Regulatory material
- Financial reports
- Internal FAQs
The application can then ground its response in those sources.
Step 1: Ingest the Documents
Before we can retrieve anything, we need to prepare the data.
Imagine a document repository containing thousands of PDFs and internal documents.
The ingestion pipeline could look like this:
Documents
↓
Parser
↓
Text Extraction
↓
Cleaning
↓
Chunking
↓
Metadata
↓
Embeddings
↓
Vector Database
Metadata is particularly useful.
A chunk might contain information such as:
{
"document": "international-payments-policy.pdf",
"department": "payments",
"version": "2026-04",
"access_level": "internal",
"section": "exceptions"
}
The exact metadata depends on the organization's data model.
The important part is that the retrieval system should know more than just the text itself.
Step 2: Chunk the Documents
LLMs have context limits, and sending entire documents for every query is inefficient.
So documents are usually divided into smaller chunks.
A simple approach might be:
def chunk_text(text, chunk_size=800):
words = text.split()
return [
" ".join(words[i:i + chunk_size])
for i in range(0, len(words), chunk_size)
]
This is only a basic example.
Production systems often use structure-aware chunking instead.
For example:
Document
├── Section
│ ├── Subsection
│ │ └── Paragraph
│ └── Subsection
└── Section
Preserving document structure can improve retrieval because related content stays together.
Step 3: Generate Embeddings
Each chunk can be converted into an embedding.
Conceptually:
embedding = embedding_model.encode(chunk)
The resulting vector represents the semantic characteristics of the text.
For example:
"international payment exception process"
↓
[0.12, -0.31, 0.77, ...]
The vector is then stored in a vector database along with the original text and metadata.
When a user asks a question, the question can also be embedded.
The system can then search for vectors that are semantically similar.
Step 4: Retrieve Relevant Context
Suppose the user asks:
"How do we handle exceptions for international transfers?"
A semantic search might retrieve chunks related to:
- International transfers
- Payment exceptions
- Approval workflows
- Escalation procedures
The application then selects the most relevant results.
A simple retrieval function could look like:
def retrieve_documents(query, vector_db, top_k=5):
query_vector = embedding_model.encode(query)
results = vector_db.search(
query_vector,
top_k=top_k
)
return results
In a real application, retrieval usually needs more than vector similarity.
Useful additions can include:
- Keyword search
- Metadata filtering
- Hybrid search
- Re-ranking
- Access-control filtering
Permission-Aware Retrieval
This is one of the most important parts of a banking RAG system.
Imagine that the knowledge base contains documents for:
- Customer service
- Risk
- Compliance
- Finance
- Operations
An employee shouldn't automatically receive every document just because the vector search considers it relevant.
Instead, permissions should influence retrieval.
Conceptually:
results = vector_db.search(query_vector, top_k=20)
authorized = [
document
for document in results
if user_can_access(user, document)
]
context = authorized[:5]
This example is intentionally simplified.
In production, authorization should be designed as part of the application's security architecture rather than treated as a small filtering function.
The important idea is:
Relevant does not necessarily mean authorized.
Step 5: Build the LLM Context
Once relevant and authorized documents have been retrieved, we can construct the prompt.
For example:
context = "\n\n".join(
document["text"]
for document in authorized_documents
)
prompt = f"""
Answer the user's question using only the provided context.
If the context does not contain enough information,
say that the information is unavailable.
Context:
{context}
Question:
{user_question}
"""
The LLM can then generate the response using the retrieved information.
A production prompt would typically include additional instructions and safeguards.
Step 6: Generate the Response
The application sends the constructed request to the selected model.
Conceptually:
response = llm.generate(prompt)
The final application response might look like:
The current process requires the payment exception
to be reviewed by the Payments Operations team.
Source:
International Payments Policy — Section 4.2
Showing sources can make the system easier to review and debug.
It also gives users a way to verify where the answer came from.
RAG Does Not Automatically Prevent Hallucinations
It's tempting to think:
"We added RAG, so the chatbot won't hallucinate."
That's not quite true.
There are multiple failure points.
For example:
Wrong Document
↓
Wrong Retrieval
↓
Wrong Context
↓
Wrong Answer
Even if retrieval works correctly, the model can still misunderstand the supplied context.
That's why evaluation needs to cover the complete pipeline.
Useful metrics and checks can include:
- Retrieval relevance
- Context relevance
- Answer correctness
- Citation accuracy
- Unsupported claims
- Response latency
- User feedback
A good test set should include both easy and difficult questions.
Hybrid Search Can Be Useful
Vector search isn't always the best solution by itself.
Banking documents often contain exact identifiers, product codes, policy numbers, abbreviations, and technical terminology.
Consider:
"Policy PYM-2026-17"
An exact keyword search may be extremely useful here.
But a user might ask:
"What is the procedure for handling payment exceptions?"
without using the exact terminology in the document.
This is where hybrid retrieval can help.
User Query
│
┌───────┴────────┐
↓ ↓
Keyword Search Vector Search
│ │
└───────┬────────┘
↓
Re-ranking
↓
Relevant Context
The exact implementation depends on the search infrastructure being used.
Connecting RAG to Banking APIs
Not every question can be answered from documents.
Some questions require live information.
For example:
"What is the status of this customer's application?"
That's fundamentally different from:
"What is the policy for application verification?"
The first may require access to a live business system.
A safer architecture is to expose controlled APIs or functions rather than giving the LLM unrestricted database access.
LLM
↓
Tool / Function
↓
API Gateway
↓
Business Service
↓
Authorized Data
For example:
def get_application_status(application_id, user):
authorize(user)
return banking_api.get_status(application_id)
The AI can request the operation, but the underlying application remains responsible for authentication and authorization.
Don't Give the LLM Direct Database Access
It might be tempting to build:
LLM → SQL Database
But this can create unnecessary security and governance problems.
A controlled service layer is usually easier to reason about:
LLM
↓
Approved Tool
↓
Business Logic
↓
Database
The service can restrict:
- Which tables can be accessed
- Which fields can be returned
- Which operations are allowed
- Which users can perform them
This also creates a better audit boundary.
Observability for AI Applications
Traditional application monitoring isn't enough for RAG.
A normal API might track:
Latency
Errors
Traffic
CPU
Memory
A RAG application should also track:
Retrieved documents
Retrieval score
Context size
LLM latency
Token usage
Model response
Citation accuracy
User feedback
Suppose a user reports:
"The assistant gave me the wrong policy."
An engineer should be able to trace:
Question
↓
Retrieved Documents
↓
Applied Filters
↓
Context
↓
Prompt
↓
Model
↓
Response
Without this information, debugging AI systems can become extremely difficult.
Start With a Small Use Case
A common mistake is trying to build an AI assistant that can answer every banking question from day one.
A better engineering approach is to define a narrow initial scope.
For example:
Version 1: Internal policy search
The assistant only answers questions using an approved collection of documents.
Then the team can evaluate:
- Retrieval quality
- Security
- Response accuracy
- Latency
- User feedback
Once the basic architecture works, additional data sources and workflows can be introduced.
Example Production Architecture
Putting the components together:
┌──────────────┐
│ User │
└──────┬───────┘
│
▼
┌────────────────┐
│ Web / Mobile UI│
└───────┬────────┘
│
▼
┌────────────────┐
│ Application API│
└───────┬────────┘
│
┌────────────┴────────────┐
│ │
▼ ▼
┌───────────────┐ ┌────────────────┐
│ Auth / Access │ │ Banking APIs │
│ Control │ │ & Services │
└───────┬───────┘ └────────────────┘
│
▼
┌───────────────┐
│ Retrieval │
│ Layer │
└───────┬───────┘
│
┌───────┴────────┐
▼ ▼
┌─────────────┐ ┌──────────────┐
│ Vector DB │ │ Keyword/ │
│ │ │ Hybrid Search│
└──────┬──────┘ └──────┬───────┘
│ │
└───────┬────────┘
▼
┌───────────────┐
│ Re-ranking / │
│ Context Build │
└───────┬───────┘
│
▼
┌─────────┐
│ LLM │
└────┬────┘
│
▼
┌───────────────┐
│ Validation / │
│ Guardrails │
└───────┬───────┘
│
▼
Final Answer
This separation makes it easier to evolve individual components without turning the entire application into one tightly coupled system.
RAG vs Fine-Tuning
RAG and fine-tuning are sometimes treated as competing approaches, but they solve different problems.
RAG is useful when the application needs access to external or changing information.
Fine-tuning can be useful when the goal is to modify model behavior using examples.
A simple way to think about it:
| Requirement | RAG | Fine-Tuning |
|---|---|---|
| Frequently changing knowledge | ✓ | Not ideal |
| Private enterprise documents | ✓ | Not the primary purpose |
| Grounding answers in documents | ✓ | Not the primary purpose |
| Response style | Limited | ✓ |
| Task-specific behavior | Limited | ✓ |
| Easy knowledge updates | ✓ | Requires another training/update process |
For some applications, both techniques can be used together.
Where Betadrix Comes In
The engineering challenges around banking AI go beyond selecting an LLM. Building these systems can involve application development, RAG pipelines, enterprise integrations, security controls, and deployment architecture.
Betadrix has also published a technical overview of generative AI solutions for banking, covering RAG pipelines, LLM integration, security, and deployment considerations.
The goal should not be to add an LLM to an existing application and call it an AI platform.
The goal is to design an application where the model operates within clearly defined technical and security boundaries.
Final Thoughts
A production banking AI assistant is essentially a distributed software system with an LLM at its center—not a chatbot with a database attached.
The key engineering pieces are:
- Reliable document ingestion
- Good chunking and metadata
- Semantic and hybrid retrieval
- Permission-aware access
- Controlled API integrations
- Grounded generation
- Evaluation and observability
- Security and governance
RAG provides a practical foundation for connecting language models with enterprise knowledge, but retrieval alone isn't enough.
The quality of the final system depends on the entire pipeline—from the source data and authorization layer to retrieval, context construction, model generation, and post-response validation.
For developers building AI systems in regulated or data-sensitive environments, that architecture is often more important than the choice of LLM itself.
Top comments (0)