Deploying LLM Models on Cloud Platforms: A Step-by-Step Guide

Deploying large language models in production requires choosing between operational control and inference efficiency. Teams can self-host on raw cloud compute, orchestrate containers on Kubernetes, or consume a managed inference API. Each path involves distinct trade-offs in latency, cost structure, and maintenance overhead. This guide walks through the practical steps for each approach and shows where a request-based inference layer can eliminate infrastructure complexity.

Evaluating Deployment Paradigms

Most production LLM deployments fall into three categories. Self-hosted instances give you full control over weights, networking, and scheduling. Kubernetes-based orchestration adds autoscaling and service discovery, but introduces node management and GPU operator complexity. Managed inference APIs abstract away hardware entirely, exposing a standard HTTP interface that accepts prompts and returns completions.

The right choice depends on your throughput requirements, context window sizes, and whether your workloads are steady or bursty. For agentic pipelines that issue many long-context requests, the cost model of your provider matters as much as the hardware.

Self-Hosting on Raw Cloud Compute

If you need to run fine-tuned weights or comply with strict data residency rules, provisioning dedicated GPU instances is the starting point. On AWS, that means launching a P4d or G6e instance, installing NVIDIA drivers and the CUDA toolkit, then pulling a model serving framework such as vLLM or Text Generation Inference.

A minimal vLLM deployment with Docker looks like this:

docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 8000:8000 \
  vllm/vllm-openai:latest \
  --model meta-llama/Llama-3.3-70B-Instruct \
  --tensor-parallel-size 8 \
  --max-model-len 32768

You then expose port 8000 through a load balancer and manage TLS termination yourself. Scaling this to multiple nodes requires either manual replication or an orchestration layer. You also pay for idle GPU time, which makes this approach expensive for sporadic traffic.

Orchestrating with Kubernetes

Kubernetes adds autoscaling and health checks, but GPU workloads require extra preparation. You need GPU-enabled node pools, the NVIDIA device plugin, and a cluster autoscaler that understands GPU constraints. Model weights are typically stored in object storage and mounted at runtime, or baked into container images if your registry supports large layers.

A simplified deployment manifest for a model server might look like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-inference
spec:
  replicas: 1
  selector:
    matchLabels:
      app: llm-inference
  template:
    metadata:
      labels:
        app: llm-inference
    spec:
      nodeSelector:
        cloud.google.com/gke-accelerator: nvidia-l4
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        args:
          - --model
          - meta-llama/Llama-3.3-70B-Instruct
        resources:
          limits:
            nvidia.com/gpu: "4"
        ports:
        - containerPort: 8000

You will still need to configure Horizontal Pod Autoscaling based on custom GPU metrics, manage spot instance interruptions, and optimize KV-cache memory usage to prevent out-of-memory errors during long-context requests.

Managed Inference with Oxlo.ai

If your team prioritizes shipping features over patching CUDA drivers, a managed inference API removes the infrastructure surface area entirely. Oxlo.ai offers a developer-first platform with flat per-request pricing, meaning one fixed cost per API call regardless of prompt length. This structure is particularly effective for long-context and agentic workloads, where token-based billing can otherwise dominate your budget.

Oxlo.ai hosts 45+ open-source and proprietary models, including Llama 3.3 70B, DeepSeek R1 671B MoE, Qwen 3 32B, and Kimi K2.6. There are no cold starts on popular models, and the API is fully compatible with the OpenAI SDK. Switching your existing client requires only a base URL change.

Here is a drop-in Python example:

from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="your_oxlo_api_key"
)

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {"role": "system", "content": "You are a precise technical assistant."},
        {"role": "user", "content": "Explain the trade-offs between tensor parallelism and pipeline parallelism when serving a 70B model."}
    ],
    temperature=0.2,
    max_tokens=1024
)

print(response.choices[0].message.content)

Because Oxlo.ai charges per request rather than per token, you can send large system prompts, multi-turn conversation histories, or retrieval-augmented context without watching metered costs scale linearly with input length. For teams building agents that iterate over long documents, this predictability simplifies budgeting and removes the incentive to truncate context for cost reasons.

You can explore request limits and plan details on the Oxlo.ai pricing page.

Cost and Workload Considerations

Self-hosting makes sense when you have steady, high-throughput traffic that saturates GPUs around the clock, or when regulatory requirements demand on-premise weights. Kubernetes improves utilization across teams, but the operational tax is significant. For most product teams, the time spent tuning GPU drivers and autoscalers is time not spent improving the application.

Managed APIs invert the model. You trade hardware control for zero-maintenance scaling and standardized billing. If your application sends variable-length prompts, includes vision inputs, or runs multi-step agent loops, a flat per-request model removes the cost unpredictability associated with token-based metering. You do not need to forecast input token ratios or cap context windows to stay inside a budget.

Security and Observability

Regardless of the deployment mode, treat your inference endpoint as production infrastructure. For self-hosted stacks, run model servers inside a private subnet and expose them only through an API gateway with rate limiting and request validation. Rotate secrets through a vault and restrict egress from GPU nodes.

For managed APIs, verify that your provider supports API key scoping, usage dashboards, and streaming responses so you can monitor latency percentiles without running your own telemetry sidecars. Oxlo.ai provides standard HTTP streaming, function calling, JSON mode, and vision endpoints, which lets you observe traffic patterns with existing OpenAI-compatible middleware.

Conclusion

Deploying LLMs on cloud platforms ranges from low-level GPU provisioning to a single API key swap. Self-hosting offers maximum control, Kubernetes adds orchestration, and managed inference APIs like Oxlo.ai eliminate infrastructure overhead. If your workloads involve long prompts, agentic tool use, or unpredictable traffic patterns, flat per-request pricing and OpenAI SDK compatibility make Oxlo.ai a strong, relevant option. Start with the managed path to validate your use case, then move to self-hosted infrastructure only when the operational cost is justified by your scale or compliance needs.