How Fintech Giants Like Revolut Are Winning with On‑Premise LLMs (2026 Guide)
Introduction
Fintechs are cutting ties with cloud‑only AI and moving their conversational models inside the data center—today. Revolut, N26, and Wise (formerly TransferWise) have already deployed on‑premise large language models (LLMs) to satisfy GDPR‑level privacy, shave latency below 50 ms, and slash token costs by up to 70 %.
If you’re a CTO or platform engineer wondering how to replicate their success, this guide walks you through a production‑ready, end‑to‑end deployment—from picking the right open‑source model to wiring monitoring, security, and CI/CD. All examples are ready‑to‑run on your own hardware.
1. Market Momentum (Quick Numbers)
| Metric (last 18 mo) | Value |
|---|---|
| Google Trends “self‑hosted LLM” | +215 % YoY |
| “LLM on‑prem” searches | +178 % YoY |
| Production‑ready models released | Llama‑3, Gemma‑2, Falcon‑180B |
| Fintechs reporting on‑prem LLMs (2024‑2025) | 12 + major players |
The combination of mature MLOps tooling (MLflow, DeepSpeed, TGI) and cheaper H100‑class GPUs makes the economics compelling.
2. Choosing the Right Model
| Model | Parameters | Recommended HW* | Accuracy (MMLU) | Typical Cost @ Scale |
|---|---|---|---|---|
| Gemma‑2‑27B | 27 B | 1 × NVIDIA H100 80 GB (2‑way TP) | 78 % of GPT‑3.5 | $0.12 / 1 M tokens (est.) |
| Llama‑3‑13B | 13 B | 1 × H100 80 GB (1‑way TP) | 75 % of GPT‑3.5 | $0.09 / 1 M tokens |
| Falcon‑180B | 180 B | 8‑12 × H100 80 GB (ZeRO‑3) | 84 % of GPT‑4‑mini | $0.45 / 1 M tokens |
*HW recommendation assumes FP8 inference with NVIDIA TensorRT‑LLM.
Fintech recommendation: start with Gemma‑2‑27B for a single‑node proof‑of‑concept; scale to Falcon‑180B only if you need massive throughput (e.g., 10 k RPS for fraud‑prevention chat).
3. Architecture Overview
+-------------------+ +-------------------+ +-------------------+
| Client Apps | <---> | API Gateway | <---> | Inference Service|
| (Web, Mobile) | | (Envoy/NGINX) | | (TGI + TorchServe)|
+-------------------+ +-------------------+ +-------------------+
| | |
| | |
v v v
+-------------------+ +-------------------+ +-------------------+
| Auth / Rate‑Lim | | Monitoring Stack| | Model Store (HF) |
| Limiter (OPA) | | (Prometheus) | | (S3/MinIO) |
+-------------------+ +-------------------+ +-------------------+
- API Gateway – Envoy with JWT validation, rate‑limiting via Open Policy Agent (OPA).
- Inference Service – HuggingFace Text Generation Inference (TGI) wrapped by TorchServe for multi‑model support.
- Model Store – Private MinIO bucket; models are pulled at container start.
- Monitoring – Prometheus scrapes GPU metrics; Grafana dashboards show latency, token‑rate, and cost.
4. Step‑by‑Step Deployment
4.1 Prerequisites
# OS: Ubuntu 22.04 LTS
# Docker Engine 24.x, Docker‑Compose 2.20
# NVIDIA driver ≥ 550, nvidia‑container‑toolkit
# 1× H100 80 GB (or equivalent)
4.2 Pull and Convert the Model
# 1️⃣ Create a private MinIO bucket (replace <access> <secret>)
mc alias set myminio http://minio.local:9000 <access> <secret>
mc mb myminio/models
# 2️⃣ Download Gemma‑2‑27B from HF (requires auth token)
export HF_TOKEN=hf_XXXXXXXXXXXXXXXX
git lfs install
git clone https://huggingface.co/google/gemma-2-27b
# 3️⃣ Convert to TensorRT‑LLM format (FP8)
cd gemma-2-27b
python3 convert_to_trt.py \
--model-dir . \
--output-dir trt_engine \
--dtype fp8 \
--max-batch-size 8
# 4️⃣ Upload engine to MinIO
mc cp -r trt_engine myminio/models/gemma-2-27b/
4.3 Docker‑Compose Stack
# docker-compose.yml
version: "3.9"
services:
api-gateway:
image: envoyproxy/envoy:v1.28-latest
ports: ["8080:8080"]
volumes:
- ./envoy.yaml:/etc/envoy/envoy.yaml
depends_on: [inference]
inference:
image: ghcr.io/huggingface/text-generation-inference:0.10.0
environment:
- MODEL_ID=gemma-2-27b
- HF_TOKEN=${HF_TOKEN}
- MAX_BATCH_SIZE=8
- PORT=80
runtime: nvidia
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
command: >
--model-id /models/gemma-2-27b
--max-batch-size ${MAX_BATCH_SIZE}
--trust-remote-code
volumes:
- ./models:/models:ro
Start the stack:
docker compose up -d
4.4 Kubernetes (Production)
# k8s/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
name: fintech-llm
---
# k8s/gemma-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: gemma-inference
namespace: fintech-llm
spec:
replicas: 1
selector:
matchLabels:
app: gemma
template:
metadata:
labels:
app: gemma
spec:
containers:
- name: tgi
image: ghcr.io/huggingface/text-generation-inference:0.10.0
args:
- --model-id=/models/gemma-2-27b
- --max-batch-size=8
- --trust-remote-code
resources:
limits:
nvidia.com/gpu: 1
volumeMounts:
- name: model-store
mountPath: /models
volumes:
- name: model-store
persistentVolumeClaim:
claimName: gemma-pvc
---
# k8s/gemma-service.yaml
apiVersion: v1
kind: Service
metadata:
name: gemma-svc
namespace: fintech-llm
spec:
selector:
app: gemma
ports:
- protocol: TCP
port: 80
targetPort: 80
type: ClusterIP
Apply:
kubectl apply -f k8s/
5. Monitoring & Cost Estimation
5.1 Prometheus Exporter (Python)
# gpu_exporter.py
import time, subprocess, os
from prometheus_client import start_http_server, Gauge
GPU_UTIL = Gauge('gpu_util_percent', 'GPU utilization')
GPU_MEM = Gauge('gpu_memory_used_bytes', 'GPU memory used')
def collect():
out = subprocess.check_output(['nvidia-smi',
'--query-gpu=utilization.gpu,memory.used',
'--format=csv,noheader,nounits']).decode()
util, mem = map(int, out.strip().split(','))
GPU_UTIL.set(util)
GPU_MEM.set(mem * 1024 * 1024)
if __name__ == '__main__':
start_http_server(9100)
while True:
collect()
time.sleep(5)
Deploy as a sidecar or DaemonSet; Prometheus scrapes http://<node>:9100/metrics.
5.2 Cost Dashboard (Grafana)
Create a panel with the expression:
sum(rate
---
*Herramienta mencionada: [Groq Cloud](https://groq.com)*
Top comments (0)