Running large language models on-premise gives you full control over data residency, latency, and model weights. It also shifts the burden of GPU provisioning, driver management, and continuous optimization onto your team. This guide walks through a production-ready deployment pipeline using open-source tooling. If your priority is shipping features rather than maintaining infrastructure, a managed inference platform like Oxlo.ai can eliminate most of this complexity while preserving full API compatibility.
Hardware and Environment Planning
Start by auditing your hardware topology. A single Llama 3.3 70B instance in FP16 requires roughly 140 GB of VRAM, which means at least two NVIDIA A100 80 GB GPUs or three A100 40 GB GPUs with tensor parallelism. For larger dense models or massive mixture-of-experts checkpoints like DeepSeek R1 671B MoE, you will need multi-node GPU clusters with high-bandwidth interconnects such as InfiniBand or NVLink. Map out your PCIe bandwidth, NUMA topology, and network backplane before you install drivers. If this level of hardware orchestration is not core to your business, Oxlo.ai hosts these exact models on dedicated infrastructure with no cold starts, letting you route requests via a standard OpenAI-compatible client without touching a driver.
Model Selection and Acquisition
Download weights from Hugging Face or a vendor portal. Verify checksums and license terms. Popular open-source choices include:
- General-purpose reasoning: Llama 3.3 70B, Qwen 3 32B, GLM 5
- Deep reasoning and coding: DeepSeek R1 671B MoE, DeepSeek V4 Flash, Kimi K2.6
- Efficiency-focused: DeepSeek V3.2, Minimax M2.5
Store weights on high-speed NVMe and restrict filesystem permissions. Keep in mind that serving a model is only half the battle. You must also patch implementations as new safetensor formats, attention optimizations, and context-length extensions are released. Oxlo.ai maintains 45+ open-source and proprietary models across seven categories, from vision to audio to embeddings, so you can switch between Llama, DeepSeek, Kimi, and Qwen families without re-downloading terabytes of weights.
Serving Infrastructure
For throughput and OpenAI-compatible endpoints, vLLM is the current production standard. It implements PagedAttention to minimize KV-cache waste and exposes a /v1/chat/completions route that mirrors the OpenAI spec. A minimal Docker launch for Llama 3.3 70B on four GPUs looks like this:
docker run --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-p 8000:8000 \
vllm/vllm-openai:latest \
--model meta-llama/Llama-3.3-70B-Instruct \
--tensor-parallel-size 4 \
--max-model-len 32768 \
--dtype autoAlternatives such as TGI and TensorRT-LLM trade ecosystem flexibility for lower latency, but they require custom model compilation and stricter version pinning. Whichever engine you choose, pin your container digests and test rolling restarts. Upgrading a serving engine on a Friday evening because a new CUDA driver broke compatibility is a common on-premise hazard. With Oxlo.ai, the runtime, driver stack, and model format migrations are handled upstream. You call https://api.oxlo.ai/v1 and receive a streaming response identical to the local vLLM layout.
Containerization and Orchestration
Once your single-node proof of concept is stable, move to Kubernetes with the NVIDIA GPU Operator. Create device-plugin daemonsets, set resource limits, and use node selectors to isolate LLM workloads from training jobs. A typical deployment manifest will request nvidia.com/gpu: 4 and mount a ReadWriteMany PVC for model weights if you run multiple replicas. Use Helm to template your inference server, and keep your liveness and readiness probes lightweight. A failing health check against a 70B parameter model can trigger a 10-minute restart cycle.
API Gateway and Load Balancing
Expose your inference pods through an API gateway such as Envoy or NGINX. Implement token bucket rate limiting, request size caps, and API key rotation. You will also need request
Top comments (0)