Deploying LLM Models on Cloud: A Comprehensive Guide

Running large language models in production requires more than a GPU. You need to handle model weights, serving frameworks, scaling logic, and cost controls. Cloud infrastructure gives you the flexibility to deploy near your users, but the path from a downloaded checkpoint to a reliable endpoint is full of operational detail. This guide walks through both self-hosted and managed approaches, with concrete code and architecture notes you can use today.

Why Cloud Deployment Matters for LLMs

On-premise clusters tie you to hardware refresh cycles and physical limits. Cloud platforms let you provision A100s, H100s, or L4s in minutes, scale to zero or across regions, and experiment with different instance types without capital expenditure. For LLMs, this elasticity matters because inference traffic is rarely uniform. Batch workloads, chat applications, and agentic pipelines all have different latency and throughput requirements.

Self-Hosted vs. Managed Inference

Self-hosting gives you full control over weights, data residency, and request routing. It is often the right choice when you have steady, predictable traffic and strict compliance requirements. Managed inference APIs remove infrastructure overhead, eliminate cold starts on popular models, and give you immediate access to a broad model catalog. If your team is spending more time debugging CUDA drivers than shipping product features, the managed route deserves a hard look.

Core Components of a Self-Hosted Stack

A minimal production stack includes four layers. First, model weights stored in an object store or on a mounted volume. Second, a model server that handles tokenization, batching, and generation. Third, a container runtime such as Docker to package dependencies. Fourth, an orchestrator like Kubernetes to handle scheduling, health checks, and autoscaling. You will also want a metrics pipeline, because GPU utilization and queue depth are the signals that drive scaling decisions.

Deploying an LLM with vLLM and Docker

vLLM is a popular open-source server that implements PagedAttention for efficient KV-cache management. To run Llama 3.3 70B on a multi-GPU host, pull the official image and set tensor parallelism.

docker run --gpus all \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  --model meta-llama/Llama-3.3-70B-Instruct \
  --tensor-parallel-size 2 \
  --dtype bfloat16

This exposes an OpenAI-compatible chat endpoint at http://localhost:8000/v1/chat/completions. For production, pin the image tag, add a reverse proxy with rate limiting, and mount your Hugging Face token as a secret rather than relying on the cache directory.

Orchestration with Kubernetes

A single container is not fault tolerant. In Kubernetes, you can represent the deployment as follows.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-inference
spec:
  replicas: 1
  selector:
    matchLabels:
      app: llm-inference
  template:
    metadata:
      labels:
        app: llm-inference
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        ports:
        - containerPort: 8000
        resources:
          limits:
            nvidia.com/gpu: 2
        command:
        - python3
        - -m
        - vllm.entrypoints.openai.api_server
        - --model
        - meta-llama/Llama-3.3-70B-Instruct
        - --tensor-parallel-size
        - "2"

Add a ClusterIP service and a HorizontalPodAutoscaler if you have multiple model replicas. Keep in mind that LLM pods do not scale as fast as stateless web servers. Preload weights onto a shared volume or use a model cache sidecar to reduce startup time.

Optimization and Cost Control

Inference costs come from GPU time, not just wall-clock time. Three techniques reduce spend without sacrificing output quality. Quantization, such as AWQ or GPTQ, shrinks model weights and increases tokens per second. Continuous batching, built into vLLM and TGI, keeps the GPU saturated across concurrent requests. Prompt caching avoids reprocessing identical system prompts across multiple calls. Even with these optimizations, self-hosted clusters often run at low utilization during off-peak hours. That idle time is pure overhead.

The Managed Alternative

If you need access to many models, variable traffic, or immediate multi-region deployment, maintaining your own fleet is a distraction. Managed providers handle loading, routing, and scaling. Oxlo.ai is a developer-first inference platform that offers 45+ open-source and proprietary models across seven categories, including reasoning, code, vision, image generation, audio, embeddings, and object detection. Flagship models such as DeepSeek R1 671B MoE, Qwen 3 32B, Kimi K2.6, and Llama 3.3 70B are available with no cold starts. Because Oxlo.ai exposes a fully OpenAI-compatible API, you can point existing client code at a new base URL without rewriting request logic.

Integrating Oxlo.ai

Switching from a self-hosted endpoint to Oxlo.ai requires only a configuration change. Below is a Python example using the official OpenAI SDK.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"],
)

response = client.chat.completions.create(
    model="Llama-3.3-70B",
    messages=[
        {"role": "system", "content": "You are a precise technical assistant."},
        {"role": "user", "content": "Explain the trade-offs between tensor parallelism and pipeline parallelism."},
    ],
    stream=False,
)

print(response.choices[0].message.content)

Oxlo.ai also supports streaming, function calling, JSON mode, vision inputs, and multi-turn conversations. If your application already uses the OpenAI SDK, Node.js client, or cURL, the integration is a drop-in replacement.

Pricing Model and Workload Fit

Most managed inference providers bill by the token. For short queries, this is predictable. For long-context workloads, agentic loops, or large system prompts, token-based costs scale linearly with input length and can become a dominant line item. Oxlo.ai uses request-based pricing: one flat cost per API request regardless of prompt length. This makes it significantly cheaper for workloads that pass large contexts or run multi-step agent workflows. For current plan details, see https://oxlo.ai/pricing. Oxlo.ai offers a free tier with 60 requests per day and a 7-day full-access trial, so you can validate workload fit before committing.

Conclusion

Deploying LLMs on the cloud is a spectrum. On one end, you self-host with Docker and Kubernetes for maximum control. On the other, you consume a managed API and focus on product logic. Self-hosting wins when you have steady traffic, custom hardware requirements, or strict data residency needs. Managed APIs win when you value elasticity, broad model choice, and fast integration. For teams building long-context applications, agentic pipelines, or multi-modal products, Oxlo.ai is a genuinely relevant option that removes infrastructure overhead while keeping costs predictable.