Originally published on the HostingPodium Enterprise Engineering Blog.
Free SRE & Developer Diagnostics: Test your stack live with our Rich Snippets Generator & Schema.org JSON-LD Creator, Free Website Speed & Core Web Vitals Checker, or explore all 9 Free Webmaster & Cloud Engineering Tools.
This guide details a production-grade Enterprise RAG architecture, from data ingestion to LLM inference, with concrete configuration and runbook procedures. It emphasizes isolation, observability, and predictable latency at scale.
Scope and Terminology
This guide defines the reference architecture for enterprise Retrieval-Augmented Generation (RAG) systems that must satisfy multi-tenant isolation, sub-second latency, and compliance-driven data handling. All components are expressed as containerized services orchestrated by Kubernetes 1.28+ with strict resource quotas and pod security policies.
Core Building Blocks
- Ingestion Pipeline: Apache Kafka 3.5 for ordered event streaming, Confluent Schema Registry for Avro enforcement.
- Document Store: PostgreSQL 15 with pgvector extension (dim=768, ivfflat index, nlist=1024).
- Vector Index: Milvus 2.4 deployed in a StatefulSet with 3 replicas, using IVF_FLAT index and GPU acceleration (NVIDIA A100, 40 GB).
- Retrieval Service: FastAPI 0.109, uvicorn workers=4, gunicorn timeout=30, deployed behind Envoy proxy.
- LLM Inference: vLLM 0.3.0 on TensorRT-optimized LLaMA-2-70B, batch size=8, max_tokens=512.
- Orchestration: Argo Workflows for batch jobs, Tekton pipelines for CI/CD.
Data Ingestion Pattern
Documents arrive via HTTPS POST to /ingest. The endpoint validates JWT claims against the corporate IdP, then produces a Kafka record keyed by tenant ID. A consumer group (size=6) runs a Python worker that extracts text, normalizes Unicode (NFKC), and writes raw blobs to an S3-compatible bucket with SSE-KMS encryption. Metadata is persisted to PostgreSQL within a single transaction to guarantee referential integrity.
# ingest_worker.py
import json, boto3, psycopg2, kafka
from kafka import KafkaConsumer
consumer = KafkaConsumer('doc-ingest', group_id='ingest-workers', bootstrap_servers='kafka:9092')
for msg in consumer:
payload = json.loads(msg.value)
tenant = payload['tenant']
s3.put_object(Bucket='enterprise-docs', Key=f"{tenant}/{payload['doc_id']}.pdf", Body=payload['blob'], ServerSideEncryption='aws:kms')
cur = pg_conn.cursor()
cur.execute("INSERT INTO documents (id, tenant, s3_key, meta) VALUES (%s,%s,%s,%s)",
(payload['doc_id'], tenant, f"{tenant}/{payload['doc_id']}.pdf", json.dumps(payload['meta'])))
pg_conn.commit()
Vectorization and Indexing
After ingestion, a Tekton task triggers a Python microservice that loads the document, runs SentenceTransformers (model=all-mpnet-base-v2) on a dedicated GPU node, and stores the 768-dimensional embedding in pgvector. The service then pushes the embedding to Milvus using the bulk insert API, which merges into the IVF_FLAT index within 200 ms per 1 k vectors.
# vectorize_and_store.py
import torch, milvus, psycopg2
model = torch.hub.load('sentence-transformers/all-mpnet-base-v2')
emb = model.encode(text)
pg_cur.execute("INSERT INTO embeddings (doc_id, vec) VALUES (%s, %s)", (doc_id, emb.tolist()))
milvus_client.insert(collection='docs', vectors=[emb], ids=[doc_id])
Retrieval Service API
The retrieval endpoint accepts a query string and optional filters (tenant, date range, tags). It performs the following steps:
- Encode the query with the same transformer model.
- Issue a
searchcall to Milvus withtop_k=10,metric_type=L2, andparams={'nprobe':32}. - Join the resulting IDs with PostgreSQL to fetch metadata and S3 URLs.
Latency breakdown (average over 5 k QPS):
| Stage | Mean (ms) | 99th %ile (ms) |
|---|---|---|
| Encode | 12 | 18 |
| Milvus search | 45 | 62 |
| Postgres join | 8 | 12 |
Generation Service Configuration
vLLM runs with the following environment variables to enforce resource caps and avoid OOM:
export VLLM_GPU_MEMORY_UTILIZATION=0.85
export VLLM_MAX_MODEL_LEN=8192
export VLLM_MAX_BATCH_SIZE=16
export VLLM_LOGGING_LEVEL=info
Each replica reserves 48 GiB of GPU memory and 64 GiB of system RAM. The pod spec sets limits.cpu: 32 and limits.memory: 96Gi. Horizontal Pod Autoscaler (HPA) targets cpu_utilization_percentage: 70 and custom_metric: request_latency_ms with a target of 120 ms.
Deployment Topology
All services reside in a dedicated namespace enterprise-rag. NetworkPolicy isolates tenant traffic to the Ingress gateway (NGINX Ingress Controller) and blocks east-west traffic that is not explicitly allowed. The diagram below outlines the flow:
Client → Ingress → Authz (OPA) → Retrieval API → Milvus ↔ PostgreSQL ↔ S3
↓
Generation API → vLLM → GPU Node
Scaling Strategy
Two scaling axes are addressed:
-
Read-heavy workloads: Increase Milvus replica count and enable sharding (partition count=4). Adjust
nlistto 4096 to keep recall > 0.92 attop_k=10. -
Generation spikes: Deploy a separate GPU node pool with node autoscaling based on
nvidia.com/gpuutilization. vLLM supports dynamic tensor parallelism; setTP_SIZE=2to split the 70B model across two GPUs.
Observability Stack
Metrics are scraped by Prometheus 2.48 using ServiceMonitors. Key SLOs:
- 99th-percentile retrieval latency ≤ 120 ms.
- Generation latency ≤ 300 ms for ≤ 256-token responses.
- GPU memory usage ≤ 85 % per replica.
Alert rules example:
ALERT RetrievalLatencyHigh
IF histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{service="retrieval"}[5m])) > 0.12
FOR 2m
LABELS {severity="critical"}
ANNOTATIONS {summary="Retrieval latency breach", description="99th-pct latency >120 ms"}
Security Controls
Tenant isolation is enforced at three layers:
- Namespace-level RBAC: each tenant receives a ServiceAccount with
role=read-onlyon its own document set. - Row-level security (RLS) in PostgreSQL: policies restrict
SELECTto rows wheretenant_id = current_setting('app.tenant'). - Object-level encryption in S3: bucket policy requires
x-amz-server-side-encryptionheader and validates KMS key ARN per tenant.
All inbound traffic terminates at the Ingress TLS termination point with certificates managed by cert-manager (ACME DNS-01). Mutual TLS is required for inter-service calls; each pod mounts a short-lived client certificate from the SPIFFE workload API.
Operational Runbook
1. Deploy New Tenant
- Create namespace
tenant-{{id}}with resource quota (CPU=200, Memory=400Gi, GPU=2). - Generate K8s ServiceAccount and bind to
tenant-{{id}}-role. - Provision KMS key via Terraform and attach to S3 bucket policy.
- Run
helm upgrade --install rag-{{id}} ./chart --set tenant.id={{id}}.
2. Incident – Retrieval Latency Spike
- Check Prometheus alert
RetrievalLatencyHigh. - Run
kubectl top pod -n enterprise-ragto verify CPU saturation. - If CPU > 90 %, increase HPA maxReplicas by 2 and observe scaling for 5 min.
- If Milvus node-level metrics show high
search_queue_length, add a replica:kubectl scale statefulset milvus -n enterprise-rag --replicas=5.
3. Patch LLM Model
- Upload new model tarball to internal artifact registry.
- Update Deployment env
VLLM_MODEL_PATHand rollout:kubectl rollout restart deployment/vllm -n enterprise-rag. - Validate
kubectl logs -f $(kubectl get pod -l app=vllm -n enterprise-rag -o name)formodel_load_time_ms
Trade-off Matrix
| Dimension | Option A: Single-GPU vLLM | Option B: Multi-GPU Tensor Parallel |
|---|---|---|
| Throughput (req/s) | ≈ 45 | ≈ 120 |
| GPU cost (USD/hr) | 2 × A100 | 4 × A100 |
| Latency 95th %ile | 210 ms | 130 ms |
| Complexity | Low (single pod) | High (distributed state) |
Future Extension Points
When regulatory changes require on-prem data residency, replace the S3 bucket with an OpenShift Data Foundation cluster and adjust the storage class in the PersistentVolumeClaim. For multimodal RAG, add a Vision Transformer encoder and extend the Milvus schema with a media_type field, keeping the same IVF index but increasing dim to 1024.
Free Webmaster, SRE & Cloud Diagnostic Tools (No Signup Required)
Verify and benchmark your production infrastructure directly from your browser using the HostingPodium Free Webmaster & Developer Suite:
| Free Engineering Utility | Primary Diagnostic Use Case |
|---|---|
| Global DNS Propagation & SSL Checker | Verify live A, AAAA, CNAME, MX, TXT, NS & CAA records across 21+ global resolvers + TLS certificate audits |
| Website Speed Test & Core Web Vitals Auditor | Measure server TTFB, LCP, INP, CLS, and HTTP/2 compression latency in real time |
| Rich Snippets & Schema.org JSON-LD Generator | Generate & validate structured JSON-LD markup for TechArticles, FAQs, Products & Organizations |
| HTTP Redirect Chain & Status Code Tracer | Trace 301/302 redirect hops, inspect response headers, and verify load balancer health checks |
| DMARC, SPF & DKIM Email Security Validator | Audit domain email authentication policies and generate compliant DMARC records |
| Google SERP Simulator & Meta Tag Previewer | Preview desktop/mobile Google snippets, OpenGraph cards, and pixel-width truncation |
| Free WebP Image Converter & Optimizer | Client-side PNG/JPG to WebP compression for faster Core Web Vitals LCP |
Related Enterprise Architecture & Benchmarks
- Canonical Engineering Guide: Hybrid Vector Search, Reranking & Citations in Enterprise RAG
- Managed Engineering Practice: Private AI Deployment & Enterprise RAG Engineering
- Infrastructure Benchmark Comparison: Dedicated Private AI & GPU Inference Hosting
Top comments (0)