DEV Community

Cover image for Hybrid Vector Search, Reranking & Citations in Enterprise RAG
Amir Mehmood
Amir Mehmood

Posted on Originally published at hostingpodium.com

Hybrid Vector Search, Reranking & Citations in Enterprise RAG

Originally published on the HostingPodium Enterprise Engineering Blog.

Free SRE & Developer Diagnostics: Test your stack live with our Rich Snippets Generator & Schema.org JSON-LD Creator, Free Website Speed & Core Web Vitals Checker, or explore all 9 Free Webmaster & Cloud Engineering Tools.

This guide details a production-grade Enterprise RAG architecture, from data ingestion to LLM inference, with concrete configuration and runbook procedures. It emphasizes isolation, observability, and predictable latency at scale.


Scope and Terminology

This guide defines the reference architecture for enterprise Retrieval-Augmented Generation (RAG) systems that must satisfy multi-tenant isolation, sub-second latency, and compliance-driven data handling. All components are expressed as containerized services orchestrated by Kubernetes 1.28+ with strict resource quotas and pod security policies.

Core Building Blocks

  • Ingestion Pipeline: Apache Kafka 3.5 for ordered event streaming, Confluent Schema Registry for Avro enforcement.
  • Document Store: PostgreSQL 15 with pgvector extension (dim=768, ivfflat index, nlist=1024).
  • Vector Index: Milvus 2.4 deployed in a StatefulSet with 3 replicas, using IVF_FLAT index and GPU acceleration (NVIDIA A100, 40 GB).
  • Retrieval Service: FastAPI 0.109, uvicorn workers=4, gunicorn timeout=30, deployed behind Envoy proxy.
  • LLM Inference: vLLM 0.3.0 on TensorRT-optimized LLaMA-2-70B, batch size=8, max_tokens=512.
  • Orchestration: Argo Workflows for batch jobs, Tekton pipelines for CI/CD.

Data Ingestion Pattern

Documents arrive via HTTPS POST to /ingest. The endpoint validates JWT claims against the corporate IdP, then produces a Kafka record keyed by tenant ID. A consumer group (size=6) runs a Python worker that extracts text, normalizes Unicode (NFKC), and writes raw blobs to an S3-compatible bucket with SSE-KMS encryption. Metadata is persisted to PostgreSQL within a single transaction to guarantee referential integrity.

# ingest_worker.py
import json, boto3, psycopg2, kafka
from kafka import KafkaConsumer
consumer = KafkaConsumer('doc-ingest', group_id='ingest-workers', bootstrap_servers='kafka:9092')
for msg in consumer:
 payload = json.loads(msg.value)
 tenant = payload['tenant']
 s3.put_object(Bucket='enterprise-docs', Key=f"{tenant}/{payload['doc_id']}.pdf", Body=payload['blob'], ServerSideEncryption='aws:kms')
 cur = pg_conn.cursor()
 cur.execute("INSERT INTO documents (id, tenant, s3_key, meta) VALUES (%s,%s,%s,%s)",
 (payload['doc_id'], tenant, f"{tenant}/{payload['doc_id']}.pdf", json.dumps(payload['meta'])))
 pg_conn.commit()
Enter fullscreen mode Exit fullscreen mode

Vectorization and Indexing

After ingestion, a Tekton task triggers a Python microservice that loads the document, runs SentenceTransformers (model=all-mpnet-base-v2) on a dedicated GPU node, and stores the 768-dimensional embedding in pgvector. The service then pushes the embedding to Milvus using the bulk insert API, which merges into the IVF_FLAT index within 200 ms per 1 k vectors.

# vectorize_and_store.py
import torch, milvus, psycopg2
model = torch.hub.load('sentence-transformers/all-mpnet-base-v2')
emb = model.encode(text)
pg_cur.execute("INSERT INTO embeddings (doc_id, vec) VALUES (%s, %s)", (doc_id, emb.tolist()))
milvus_client.insert(collection='docs', vectors=[emb], ids=[doc_id])
Enter fullscreen mode Exit fullscreen mode

Retrieval Service API

The retrieval endpoint accepts a query string and optional filters (tenant, date range, tags). It performs the following steps:

  • Encode the query with the same transformer model.
  • Issue a search call to Milvus with top_k=10, metric_type=L2, and params={'nprobe':32}.
  • Join the resulting IDs with PostgreSQL to fetch metadata and S3 URLs.

Latency breakdown (average over 5 k QPS):

Stage Mean (ms) 99th %ile (ms)
Encode 12 18
Milvus search 45 62
Postgres join 8 12

Generation Service Configuration

vLLM runs with the following environment variables to enforce resource caps and avoid OOM:

export VLLM_GPU_MEMORY_UTILIZATION=0.85
export VLLM_MAX_MODEL_LEN=8192
export VLLM_MAX_BATCH_SIZE=16
export VLLM_LOGGING_LEVEL=info
Enter fullscreen mode Exit fullscreen mode

Each replica reserves 48 GiB of GPU memory and 64 GiB of system RAM. The pod spec sets limits.cpu: 32 and limits.memory: 96Gi. Horizontal Pod Autoscaler (HPA) targets cpu_utilization_percentage: 70 and custom_metric: request_latency_ms with a target of 120 ms.

Deployment Topology

All services reside in a dedicated namespace enterprise-rag. NetworkPolicy isolates tenant traffic to the Ingress gateway (NGINX Ingress Controller) and blocks east-west traffic that is not explicitly allowed. The diagram below outlines the flow:

Client → Ingress → Authz (OPA) → Retrieval API → Milvus ↔ PostgreSQL ↔ S3
 ↓
 Generation API → vLLM → GPU Node
Enter fullscreen mode Exit fullscreen mode

Scaling Strategy

Two scaling axes are addressed:

  • Read-heavy workloads: Increase Milvus replica count and enable sharding (partition count=4). Adjust nlist to 4096 to keep recall > 0.92 at top_k=10.
  • Generation spikes: Deploy a separate GPU node pool with node autoscaling based on nvidia.com/gpu utilization. vLLM supports dynamic tensor parallelism; set TP_SIZE=2 to split the 70B model across two GPUs.

Observability Stack

Metrics are scraped by Prometheus 2.48 using ServiceMonitors. Key SLOs:

  • 99th-percentile retrieval latency ≤ 120 ms.
  • Generation latency ≤ 300 ms for ≤ 256-token responses.
  • GPU memory usage ≤ 85 % per replica.

Alert rules example:

ALERT RetrievalLatencyHigh
 IF histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{service="retrieval"}[5m])) > 0.12
 FOR 2m
 LABELS {severity="critical"}
 ANNOTATIONS {summary="Retrieval latency breach", description="99th-pct latency >120 ms"}
Enter fullscreen mode Exit fullscreen mode

Security Controls

Tenant isolation is enforced at three layers:

  • Namespace-level RBAC: each tenant receives a ServiceAccount with role=read-only on its own document set.
  • Row-level security (RLS) in PostgreSQL: policies restrict SELECT to rows where tenant_id = current_setting('app.tenant').
  • Object-level encryption in S3: bucket policy requires x-amz-server-side-encryption header and validates KMS key ARN per tenant.

All inbound traffic terminates at the Ingress TLS termination point with certificates managed by cert-manager (ACME DNS-01). Mutual TLS is required for inter-service calls; each pod mounts a short-lived client certificate from the SPIFFE workload API.

Operational Runbook

1. Deploy New Tenant

  • Create namespace tenant-{{id}} with resource quota (CPU=200, Memory=400Gi, GPU=2).
  • Generate K8s ServiceAccount and bind to tenant-{{id}}-role.
  • Provision KMS key via Terraform and attach to S3 bucket policy.
  • Run helm upgrade --install rag-{{id}} ./chart --set tenant.id={{id}}.

2. Incident – Retrieval Latency Spike

  • Check Prometheus alert RetrievalLatencyHigh.
  • Run kubectl top pod -n enterprise-rag to verify CPU saturation.
  • If CPU > 90 %, increase HPA maxReplicas by 2 and observe scaling for 5 min.
  • If Milvus node-level metrics show high search_queue_length, add a replica: kubectl scale statefulset milvus -n enterprise-rag --replicas=5.

3. Patch LLM Model

  • Upload new model tarball to internal artifact registry.
  • Update Deployment env VLLM_MODEL_PATH and rollout: kubectl rollout restart deployment/vllm -n enterprise-rag.
  • Validate kubectl logs -f $(kubectl get pod -l app=vllm -n enterprise-rag -o name) for model_load_time_ms

Trade-off Matrix

Dimension Option A: Single-GPU vLLM Option B: Multi-GPU Tensor Parallel
Throughput (req/s) ≈ 45 ≈ 120
GPU cost (USD/hr) 2 × A100 4 × A100
Latency 95th %ile 210 ms 130 ms
Complexity Low (single pod) High (distributed state)

Future Extension Points

When regulatory changes require on-prem data residency, replace the S3 bucket with an OpenShift Data Foundation cluster and adjust the storage class in the PersistentVolumeClaim. For multimodal RAG, add a Vision Transformer encoder and extend the Milvus schema with a media_type field, keeping the same IVF index but increasing dim to 1024.


Free Webmaster, SRE & Cloud Diagnostic Tools (No Signup Required)

Verify and benchmark your production infrastructure directly from your browser using the HostingPodium Free Webmaster & Developer Suite:

Free Engineering Utility Primary Diagnostic Use Case
Global DNS Propagation & SSL Checker Verify live A, AAAA, CNAME, MX, TXT, NS & CAA records across 21+ global resolvers + TLS certificate audits
Website Speed Test & Core Web Vitals Auditor Measure server TTFB, LCP, INP, CLS, and HTTP/2 compression latency in real time
Rich Snippets & Schema.org JSON-LD Generator Generate & validate structured JSON-LD markup for TechArticles, FAQs, Products & Organizations
HTTP Redirect Chain & Status Code Tracer Trace 301/302 redirect hops, inspect response headers, and verify load balancer health checks
DMARC, SPF & DKIM Email Security Validator Audit domain email authentication policies and generate compliant DMARC records
Google SERP Simulator & Meta Tag Previewer Preview desktop/mobile Google snippets, OpenGraph cards, and pixel-width truncation
Free WebP Image Converter & Optimizer Client-side PNG/JPG to WebP compression for faster Core Web Vitals LCP

Related Enterprise Architecture & Benchmarks

Top comments (0)