Building an Internal LLM Inference Cluster with vLLM, LiteLLM, Portainer CE, and Netdata

I’ve been working on an architecture for running LLM inference entirely inside an internal network, with a strong focus on performance, security, observability, and operational simplicity.

The serving layer is built around vLLM, with multiple GPU-backed containers serving different models such as Qwen and Llama. Each instance exposes an OpenAI-compatible API and takes advantage of features such as PagedAttention, prefix caching, and GPU acceleration.

Instead of letting applications communicate directly with individual inference servers, LiteLLM Proxy acts as the central AI gateway. It provides a single endpoint for model routing, API-key management, quotas, fallback strategies, and centralized request logging. This makes it much easier for internal applications, RAG services, AI agents, and tools such as Open WebUI to consume different models without being tightly coupled to the underlying infrastructure.

At the edge, Nginx or Caddy handles TLS termination, rate limiting, IP allowlists, and integration with SSO/OIDC, keeping the inference environment protected inside the corporate network or VPN.

For the infrastructure layer, model weights are cached locally and can be synchronized with an internal MinIO/S3 model registry, while the NVIDIA Container Toolkit provides GPU access to the Docker workloads.

The operational side is intentionally lightweight. Portainer CE provides container and stack management, with deployment configuration stored in Git for a simple GitOps-style workflow. Netdata, combined with NVIDIA/DCGM, cgroups, and application-level Prometheus metrics, provides visibility into GPU utilization, VRAM, temperature, CPU, memory, I/O, KV-cache usage, and vLLM request queues. Alerts can then be forwarded to channels such as Telegram or Slack.

The goal is not just to “run an LLM on a GPU,” but to build a practical internal inference platform that can be operated like any other production service: secured, observable, reproducible, scalable, and easy for internal systems to consume.

Architecture and implementation notes:

https://khaitrang1995.github.io/llm-internal/

LLM #vLLM #LiteLLM #AIInfrastructure #GenerativeAI #MLOps #DevOps #Docker #NVIDIA #Netdata #Portainer #RAG #AIEngineering #SystemArchitecture