Deploying LLM Models on Cloud Infrastructure: A Comprehensive Guide

#aiinfrastructure #oxlo #ai

Deploying large language models in production cloud environments requires more than provisioning a GPU instance. Engineering teams must handle model serving frameworks, autoscaling policies, quantization strategies, and continuous batching to achieve acceptable throughput and latency. This guide covers the core components of cloud LLM deployment, provides a concrete Kubernetes-based example, and explains when self-hosting is justified versus when a managed inference platform simplifies operations.

Self-Hosting vs. Managed Inference

Organizations typically choose between operating their own model serving stack or consuming models through an API. Self-hosting offers full control over hardware selection, custom quantization, and network isolation. It is often favored by teams with strict data residency requirements or those fine-tuning proprietary adapters that must remain on-premises. Managed inference, by contrast, removes the burden of driver maintenance, framework updates, and capacity planning. For many engineering teams, the decisive factor is not capability but total cost of ownership, which includes idle GPU time, engineering hours, and pricing model alignment with actual workload patterns.

Core Components of Cloud LLM Deployment

A production-ready LLM deployment stack consists of four layers.

Compute. Inference demands high-memory GPUs. A 70B parameter model in FP16 requires roughly 140 GB of VRAM, necessitating multi-GPU nodes or quantized formats such as AWQ and GPTQ.

Serving Engine. Frameworks like vLLM, TensorRT-LLM, and Text Generation Inference implement continuous batching and PagedAttention to maximize GPU utilization.

Orchestration. Kubernetes with GPU operators and node autoscalers manages pod scheduling. Karpenter or Cluster Autoscaler provisions GPU nodes on demand.

Networking and Storage. High-bandwidth NICs reduce inter-GPU communication latency. Model weights are typically stored on shared volumes or downloaded at container startup from object storage.

A Practical Deployment with vLLM

vLLM is a popular open-source serving engine that exposes an OpenAI-compatible HTTP interface. The following Kubernetes manifests deploy a Llama 3.3 70B instance with tensor parallelism across two GPUs.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-llama
  template:
    metadata:
      labels:
        app: vllm-llama
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        args:
        - --model
        - meta-llama/Llama-3.3-70B-Instruct
        - --tensor-parallel-size
        - "2"
        - --gpu-memory-utilization
        - "0.9"
        resources:
          limits:
            nvidia.com/gpu: "2"
        ports:
        - containerPort: 8000
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-service
spec:
  selector:
    app: vllm-llama
  ports:
  - port: 8000
    targetPort: 8000

This configuration requires at least two NVIDIA A100 80 GB GPUs or equivalent H100 instances. You must also ensure that the NVIDIA device plugin and GPU feature discovery are installed on the cluster. After applying the manifests, you can route traffic to the service and autoscale with Horizontal Pod Autoscaler based on GPU utilization or request queue depth.

The Hidden Costs of Self-Hosting

Token-based pricing from providers like Together AI, Fireworks AI, OpenRouter, Replicate, and Anyscale scales linearly with input length. For long-context retrieval-augmented generation or agentic loops that append extensive tool histories, costs grow unpredictably. Self-hosting shifts this to capital expenditure but introduces idle capacity. A Kubernetes cluster with reserved GPU nodes may sit underutilized outside peak hours. Additionally, popular self-hosted frameworks often incur cold starts when scaling from zero, adding latency to the first request after a quiet period.

Managed Inference with Oxlo.ai

Oxlo.ai provides a developer-first alternative that eliminates idle GPU risk and unpredictable token-based billing. Its request-based pricing charges one flat cost per API request regardless of prompt length. For long-context workloads and agentic applications that send thousands of tokens per turn, this structure can be significantly cheaper than token-based alternatives.

Oxlo.ai hosts over 45 open-source and proprietary models across seven categories, including Llama 3.3 70B, DeepSeek R1 671B MoE, Qwen 3 32B, and Kimi K2.6. The platform is fully OpenAI SDK compatible and exposes the standard base URL at https://api.oxlo.ai/v1. There are no cold starts on popular models, so latency remains consistent even after periods of inactivity.

Migrating a self-hosted or OpenAI-based client to Oxlo.ai requires only a base URL change.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="your-oxlo.ai-api-key"
)

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[{"role": "user", "content": "Explain the trade-offs between tensor parallelism and pipeline parallelism in distributed inference."}],
    stream=True
)

for chunk in response:
    print(chunk.choices[0].delta.content or "", end="")

For teams deciding between scaling out a Kubernetes GPU cluster and consuming a managed endpoint, Oxlo.ai offers a straightforward pricing page at https://oxlo.ai/pricing to compare total cost of ownership. The Free tier includes 60 requests per day across more than 16 models, with a seven-day full-access trial to evaluate production workloads.

Security and Observability

Whether self-hosted or managed, production LLM deployments require structured logging, rate limiting, and output validation. Self-hosted stacks should enforce mTLS between services, scan model artifacts for tampering, and monitor GPU memory pressure with Prometheus and Grafana. When using managed providers, verify SOC 2 compliance, review data retention policies, and implement retry logic with exponential backoff at the application layer.

Conclusion

Cloud LLM deployment is a spectrum. Teams with deep infrastructure expertise and stable, high-throughput workloads may benefit from self-hosting on dedicated GPUs using vLLM or TensorRT-LLM. For organizations prioritizing predictability, rapid iteration, and cost control on variable-length prompts, managed inference platforms reduce operational surface area. Oxlo.ai’s request-based pricing and OpenAI-compatible API make it a relevant option for long-context and agentic workloads where token-based costs would otherwise dominate the infrastructure budget. Evaluate both paths against actual request patterns, and choose the architecture that aligns engineering overhead with business requirements.

DEV Community