I Built an AI Compliance Assistant That Runs Entirely on My Own GPU — Here's What I Learned
Combining RAG, local LLM inference, Terraform-provisioned AWS infrastructure, and full-stack observability to make South African financial regulation searchable
If you've ever worked anywhere near banking compliance, you know the feeling: somewhere in a 200-page guidance note is the exact clause you need, and you have no idea which page it's on 📄🔍. South African financial institutions operate under a dense stack of regulation — AML, CTF, CPF, the FIC Act, cross-border EFT reporting, Prudential Authority requirements — and most of it lives in long, technical PDFs that don't search well.
So I decided to build something that could actually answer compliance questions, grounded in real regulatory text, running on infrastructure I control end to end 🛠️. No black-box SaaS API calls to a third-party model — the LLM inference happens locally, on my own GPU, provisioned entirely with Terraform ⚙️, and I can watch every request move through the system in real time.
This is the story of building Compliance RAG 🤖: a Retrieval-Augmented Generation platform for South African regulatory intelligence, with local GPU inference, a vector database, infrastructure-as-code, and a full observability stack behind it.
🧩 The Problem
South African banks have to navigate obligations spanning:
- Anti-Money Laundering (AML) and Counter-Terrorist Financing (CTF/CPF)
- Customer Due Diligence and risk-based approaches
- Cross-border electronic funds transfers and international reporting
- FIC Act implementation requirements
- Rules for representative offices of foreign banking institutions
A compliance officer might ask something like "What information must a bank obtain when conducting customer due diligence?" — a completely reasonable question with an answer buried somewhere in a Guidance Note. Traditional keyword search either misses the context entirely or returns ten tangentially related pages 😩. What's actually needed is a system that understands the question, retrieves the relevant passages, and answers using only what's actually in the source documents — no hallucinated legislation, no invented penalties.
💡 The Solution: Retrieval-Augmented Generation, Self-Hosted
The core idea behind RAG is simple: instead of asking a language model to answer from memory (where it might confidently make things up), you retrieve the most relevant chunks of real source documents first, and then ask the model to answer using only that retrieved context.
Here's the pipeline I built:
PDF → Document Loader → Text Splitting → Embeddings → ChromaDB
→ Similarity Search → Relevant Regulatory Context → Gemma → Answer
Regulatory PDFs — the ML/TF/PF Sector Risk Assessment, Guidance Note 7B on the FIC Act, Guidance Note 8 on EFTs, and documentation on representative offices of foreign banks — are stored in an S3 "compliance document vault" 🪣, ingested, chunked, embedded, and stored in ChromaDB for similarity search.
The regulatory document vault in S3 — each PDF is a primary source, ingested and re-embedded whenever it's updated.
When a question comes in, the system retrieves the most relevant chunks and passes them, along with the question, to a locally-hosted Google Gemma 2 2B IT model for generation. I picked Gemma 2 2B specifically because it's small enough to run comfortably on a single GPU without breaking the bank, while still being capable enough to follow strict grounding instructions — exactly the tradeoff a self-hosted compliance tool needs 🎯. Running the model locally rather than calling an external API meant provisioning real GPU infrastructure.
🏗️ Provisioning It All With Terraform
I didn't want to click around the AWS console to spin this up — I wanted it reproducible, versioned, and destroyable in one command. So every piece of AWS infrastructure — the GPU-enabled EC2 instance, the VPC, security groups, IAM roles, and the S3 bucket — is defined and deployed with Terraform 🧱.
cd infrastructure
terraform init
terraform plan
terraform apply
One terraform apply and I have a fully networked, GPU-ready EC2 instance with the right IAM permissions to talk to S3 — no manual setup, no configuration drift, and a clean terraform destroy when I don't want to pay for idle GPU time 💸.
A g5.xlarge EC2 instance with an NVIDIA A10G GPU — provisioned by Terraform, running 24/7, hosting the entire Dockerized stack.
🖥️ What It Looks Like in Practice
The frontend is a React/Vite chat interface I called the "Enterprise Regulatory Advisor." I wanted answers that weren't just fluent-sounding text — they needed to cite exactly where in the source documents they came from, so a compliance officer could go verify it themselves in seconds ✅.
Asking "What information must a bank obtain when conducting customer due diligence?" Notice the "AI Engine: Gemma-2 Active" status in the sidebar 🤖 — the model answers only from retrieved context, and — critically — flags when the source text doesn't specify something, rather than filling in the gap itself. Every answer comes with verified source references down to the page number.
That last detail mattered a lot to me. The system prompt explicitly instructs the model to avoid inventing legislation, requirements, penalties, or dates, and to say plainly when the answer isn't in the available documents. In a compliance context, a confident wrong answer is worse than no answer at all ⚠️.
📡 Watching the System Think: Observability
Running your own LLM infrastructure means you can't just trust that it's "working" — you need to actually see what's happening under the hood 🔬. I instrumented the entire FastAPI backend with OpenTelemetry, exporting traces and metrics through an OTEL Collector to Prometheus, Grafana, and Jaeger.
Prometheus querying metrics scraped from the OTEL Collector — request durations, active requests, histogram buckets, all labeled and queryable.
Prometheus gives me the raw metrics — request rates, latency histograms, active request counts 📊 — and Grafana turns those into dashboards I can actually read at a glance.
A live Grafana dashboard built on the Metrics Drilldown view — request duration buckets, throughput, and payload sizes updating in real time.
And when I need to understand why a specific request was slow — was it the embedding step, the ChromaDB similarity search, or the Gemma generation itself? — Jaeger gives me the full distributed trace 🔎.
Jaeger tracing individual API calls through the RAG pipeline, down to the microsecond.
This turned what could have been a black box into something genuinely observable — I can see request rate, latency percentiles, GPU health, and trace-level detail on any single query.
🧰 The Stack, End to End
- 🐍 Backend: Python, FastAPI, LangChain, ChromaDB, PyTorch, Hugging Face Transformers
- 🤖 AI/ML: Google Gemma 2 2B IT, local GPU inference on an NVIDIA A10G
- ⚛️ Frontend: React + Vite
- 📈 Observability: OpenTelemetry, Prometheus, Grafana, Jaeger
- ☁️ Infrastructure: AWS EC2 (GPU) + AWS S3, provisioned with Terraform, containerized with Docker Compose, with Kubernetes/Helm manifests for scaling out
- 🚀 CI/CD: GitHub Actions
Everything runs in Docker Compose for local/single-instance deployment, with a parallel Helm chart if I ever need to move this onto Kubernetes — and every underlying AWS resource traces back to a Terraform module, not a manual console click.
🎓 What I Learned
Grounding is everything. 🎯 The hardest part wasn't wiring up the pipeline — it was getting the prompt-level guardrails right so the model would refuse to speculate. A 2B parameter model is small enough to run affordably on a single GPU, but it also means you have to be deliberate about keeping it on a short leash: retrieved context in, grounded answer out, and an honest "not found" when the documents don't say.
Infrastructure-as-code pays for itself fast. 🏗️ Provisioning the GPU instance, networking, and S3 vault with Terraform meant I could tear the whole environment down between sessions and bring it back identically an hour later — a huge win for both cost control and sanity.
Observability isn't optional once you self-host. 🔭 The moment you're responsible for your own inference infrastructure, "is it working?" becomes a real question you need real tooling to answer. Watching a trace waterfall through embedding → retrieval → generation taught me more about where latency actually lives than any amount of guessing would have.
Compliance AI has a very specific bar. ⚖️ This project is explicitly not a replacement for a compliance officer or legal advisor — the disclaimer sits front and center in the docs. The goal was never "trust the AI's judgment." It was "make the source material searchable, and always show your work."
🔮 What's Next
On the roadmap: hybrid search (BM25 + vector) with cross-encoder reranking, RAGAS-based evaluation for faithfulness and retrieval precision, regulatory change detection so the vault flags when a source document gets superseded, and proper enterprise auth via Cognito/OIDC before this goes anywhere near production data.
If you're curious about the architecture or want to build something similar, the full project — Terraform configs, Helm charts, ingestion pipeline, and all — is open on GitHub. ⭐
CODE: https://github.com/limanindou/compliance-rag
YOUTUBE DEMO PC:https://youtu.be/MgLLihgKUuc?si=GoLwfbuwJBZNLQ9c
YOUTUBE DEMO MOBILE:https://youtube.com/shorts/mRaoCBRngaY?si=RFCkVGLzyi1ohKOh
Built as an AI, cloud, and compliance engineering project focused on South African regulatory intelligence 🇿🇦. Not legal advice ⚠️ — always verify against the latest official regulatory publications.
Top comments (0)