DEV Community: Pavan Madduri

Serving 3 LLMs on 1 GPU - Multi-Model Inference with Docker on OKE

Pavan Madduri — Thu, 02 Jul 2026 15:38:44 +0000

I had three small models I wanted to serve: Phi-3-mini for general chat, CodeLlama-7B for code suggestions, and a fine-tuned Mistral for document summarization. Each one fits in about 5-6GB of VRAM. An A10 GPU has 24GB. Three models, one GPU, plenty of headroom.

Running three separate vLLM deployments, each requesting a full GPU, would cost 3x and waste 18GB of VRAM. So I figured out how to serve all three from one container.

The Naive Approach (and Why It Didn't Work)

My first idea was simple: run three vLLM processes in one pod, each binding to a different port.

# Don't do this
vllm serve microsoft/Phi-3-mini-4k-instruct --port 8001 &
vllm serve codellama/CodeLlama-7b-Instruct-hf --port 8002 &
vllm serve my-org/mistral-summarizer --port 8003 &

This doesn't work because each vLLM process tries to claim the entire GPU. The second process crashes with a CUDA out-of-memory error because the first one already allocated all the VRAM.

You can set --gpu-memory-utilization 0.30 on each to split the memory, but vLLM's performance drops significantly when memory is constrained continuous batching can't work efficiently, and you lose the KV cache space that makes vLLM fast.

What Actually Works: vLLM with LoRA Adapters

If your models are fine-tuned versions of the same base model (or you can restructure them that way), vLLM supports serving multiple LoRA adapters on a single base model. One base model in GPU memory, multiple lightweight adapters loaded on demand.

docker run --gpus all -p 8000:8000 \
  -v /models:/models \
  vllm/vllm-openai:latest \
  --model /models/mistral-7b-base \
  --enable-lora \
  --lora-modules \
    "chat=/models/lora-chat" \
    "code=/models/lora-code" \
    "summary=/models/lora-summary" \
  --max-loras 3 \
  --max-model-len 4096

Clients specify which adapter to use in the model field:

# Chat model
curl http://localhost:8000/v1/chat/completions \
  -d '{"model": "chat", "messages": [...]}'

# Code model
curl http://localhost:8000/v1/chat/completions \
  -d '{"model": "code", "messages": [...]}'

# Summary model
curl http://localhost:8000/v1/chat/completions \
  -d '{"model": "summary", "messages": [...]}'

The base model (Mistral 7B) uses ~14GB VRAM. Each LoRA adapter adds only 50-200MB. All three adapters fit easily on a 24GB A10.

When You Have Different Base Models

If your models aren't LoRA variants of the same base (mine weren't originally), you have two options:

Option A: Ollama with Multiple Models

Ollama handles model loading/unloading automatically. When you request a model, it loads it into GPU memory. When memory fills up, it evicts the least recently used model.

# ollama-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ollama-multi
spec:
  replicas: 1
  template:
    spec:
      containers:
        - name: ollama
          image: ollama/ollama:latest
          ports:
            - containerPort: 11434
          env:
            - name: OLLAMA_HOST
              value: "0.0.0.0"
            - name: OLLAMA_NUM_PARALLEL
              value: "4"
          resources:
            limits:
              nvidia.com/gpu: 1
              memory: 32Gi
          volumeMounts:
            - name: models
              mountPath: /root/.ollama
      volumes:
        - name: models
          persistentVolumeClaim:
            claimName: ollama-models

Load models after deployment:

OLLAMA_IP=$(kubectl get svc ollama-multi -o jsonpath='{.spec.clusterIP}')

curl http://$OLLAMA_IP:11434/api/pull -d '{"name": "phi3:mini"}'
curl http://$OLLAMA_IP:11434/api/pull -d '{"name": "codellama:7b"}'
curl http://$OLLAMA_IP:11434/api/pull -d '{"name": "mistral:7b"}'

The downside: model swapping takes 5-15 seconds when a cold model needs to load. For a team that mostly uses one model at a time, this is fine. For concurrent usage of all three, there's latency on the first request to each model.

Option B: Triton Inference Server

NVIDIA Triton can serve multiple models on one GPU with explicit memory allocation. It's more complex to set up but gives you fine-grained control:

# model_repository/
# ├── phi3/
# │   ├── config.pbtxt
# │   └── 1/
# │       └── model.onnx
# ├── codellama/
# │   ├── config.pbtxt
# │   └── 1/
# │       └── model.onnx
# └── summarizer/
#     ├── config.pbtxt
#     └── 1/
#         └── model.onnx

FROM nvcr.io/nvidia/tritonserver:24.01-py3
COPY model_repository /models
CMD ["tritonserver", "--model-repository=/models", "--model-control-mode=explicit"]

I tried Triton and it works well for ONNX/TensorRT models. For plain HuggingFace transformer models, the conversion step adds friction. I ended up going with the LoRA approach for my use case.

My OKE Deployment

I went with vLLM + LoRA because two of my three models were fine-tuned Mistral variants anyway. I retrained the third (the code model) as a LoRA on the same Mistral base.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: multi-model-inference
  namespace: inference
spec:
  replicas: 1
  template:
    spec:
      initContainers:
        - name: model-loader
          image: ghcr.io/oracle/oci-cli:latest
          command: ["/bin/bash", "-c"]
          args:
            - |
              for model in mistral-base lora-chat lora-code lora-summary; do
                if [ ! -f /models/$model/.complete ]; then
                  oci os object bulk-download --bucket-name ai-models \
                    --prefix "models/$model/" \
                    --download-dir /models/$model \
                    --auth instance_principal
                  touch /models/$model/.complete
                fi
              done
          volumeMounts:
            - name: models
              mountPath: /models
      containers:
        - name: vllm
          image: vllm/vllm-openai:latest
          args:
            - "--model"
            - "/models/mistral-base"
            - "--enable-lora"
            - "--lora-modules"
            - "chat=/models/lora-chat"
            - "code=/models/lora-code"
            - "summary=/models/lora-summary"
            - "--max-loras"
            - "3"
            - "--max-model-len"
            - "4096"
            - "--gpu-memory-utilization"
            - "0.9"
          ports:
            - containerPort: 8000
          resources:
            limits:
              nvidia.com/gpu: 1
              memory: 32Gi
          volumeMounts:
            - name: models
              mountPath: /models
      volumes:
        - name: models
          persistentVolumeClaim:
            claimName: model-cache

Cost Impact

Setup	GPUs Needed	Monthly Cost (OCI A10)
3 separate vLLM deployments	3	$3,282
1 vLLM with 3 LoRA adapters	1	$1,094
Savings		$2,188/month

Same three models, same inference quality (LoRA adds negligible overhead), one-third the cost. The trade-off is slightly more complex deployment config and the requirement that all models share a base.

For teams exploring multi-model setups, start with Ollama (simplest), graduate to vLLM + LoRA if your models share a base, and use Triton if you need maximum control over GPU memory allocation.

Pavan Madduri - Oracle ACE Associate, CNCF Golden Kubestronaut. GitHub | LinkedIn | Website | Google Scholar | ResearchGate

Monitoring GPU Inference Containers on OKE with OpenTelemetry - What Prometheus Misses

Pavan Madduri — Thu, 02 Jul 2026 15:30:29 +0000

I had Prometheus + DCGM Exporter running on my OKE cluster. It gave me GPU utilization, memory usage, temperature. Basic stuff. What it didn't give me was the correlation between GPU metrics and inference performance request latency, tokens per second, queue depth. Two different dashboards, two different time ranges, no easy way to connect "GPU hit 95% utilization" with "p99 latency spiked to 8 seconds."

That's what led me to build otel-gpu-receiver and adopt OpenTelemetry for GPU monitoring instead of the Prometheus-only approach.

What's Wrong With DCGM Exporter Alone

DCGM Exporter is solid for hardware metrics. It gives you:

DCGM_FI_DEV_GPU_UTIL: utilization percentage
DCGM_FI_DEV_FB_USED : framebuffer memory used
DCGM_FI_DEV_GPU_TEMP: temperature
DCGM_FI_DEV_POWER_USAGE: power draw

These tell you the GPU is busy. They don't tell you why, or whether "busy" means "serving requests efficiently" or "stuck loading a model." I need application-level metrics alongside GPU metrics:

Tokens/second - actual inference throughput
Request queue depth - are requests piling up?
Time to first token - user-perceived latency
Batch size - how well is continuous batching working?

With separate Prometheus endpoints for GPU and application metrics, correlating these requires manual PromQL joins. With OpenTelemetry, everything goes through one pipeline.

The OpenTelemetry Stack on OKE

Here's what I run:

┌─────────────────────────┐
│  GPU Node               │
│  ┌───────────────────┐  │
│  │ vLLM Pod          │  │
│  │  └─ OTel SDK      │──┤──► OTel Collector ──► Backend
│  │     (traces +     │  │        (on each      (Grafana Cloud,
│  │      metrics)     │  │         node)         OCI APM, etc.)
│  └───────────────────┘  │
│  ┌───────────────────┐  │
│  │ otel-gpu-receiver │──┤──►
│  │  (NVML metrics)   │  │
│  └───────────────────┘  │
└─────────────────────────┘

1. Deploy the OTel Collector as a DaemonSet

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: otel-collector
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: otel-collector
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      containers:
        - name: collector
          image: otel/opentelemetry-collector-contrib:0.96.0
          volumeMounts:
            - name: config
              mountPath: /etc/otel
          args: ["--config=/etc/otel/config.yaml"]
      volumes:
        - name: config
          configMap:
            name: otel-collector-config

2. Collector Config - GPU + Application Metrics Together

# otel-collector-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-config
  namespace: monitoring
data:
  config.yaml: |
    receivers:
      # GPU metrics from otel-gpu-receiver
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317

      # Scrape vLLM's Prometheus metrics and convert to OTel
      prometheus:
        config:
          scrape_configs:
            - job_name: 'vllm'
              kubernetes_sd_configs:
                - role: pod
              relabel_configs:
                - source_labels: [__meta_kubernetes_pod_label_app]
                  regex: vllm
                  action: keep

    processors:
      batch:
        timeout: 10s

      # Add OKE cluster metadata to all metrics
      k8sattributes:
        auth_type: serviceAccount
        extract:
          metadata:
            - k8s.node.name
            - k8s.pod.name
            - k8s.namespace.name

    exporters:
      # Send to Grafana Cloud (or any OTel backend)
      otlphttp:
        endpoint: https://otlp-gateway-prod-us-central-0.grafana.net/otlp
        headers:
          Authorization: "Basic ${GRAFANA_TOKEN}"

      # Also export to OCI APM
      otlphttp/oci:
        endpoint: https://apm-collector.us-ashburn-1.oci.oraclecloud.com/20200101/opentelemetry
        headers:
          Authorization: "dataKey ${OCI_APM_KEY}"

    service:
      pipelines:
        metrics:
          receivers: [otlp, prometheus]
          processors: [batch, k8sattributes]
          exporters: [otlphttp, otlphttp/oci]

3. Deploy otel-gpu-receiver

This is the component I built. It reads NVIDIA GPU metrics via NVML (the same library nvidia-smi uses) and exports them as OpenTelemetry metrics:

helm install otel-gpu-receiver pmady/otel-gpu-receiver \
  --namespace monitoring \
  --set collector.endpoint="otel-collector.monitoring:4317" \
  --set scrapeInterval=15s

It runs as a DaemonSet on GPU nodes and pushes metrics to the OTel Collector on each node.

Metrics I Actually Look At

After a few weeks of running this, these are the metrics I check daily:

GPU utilization vs. tokens/second .If utilization is high but throughput is flat, something is wrong (usually a batch size issue or memory pressure).

# In Grafana overlay these two on the same panel
gpu_utilization{node="gpu-1"}
rate(vllm_generation_tokens_total{pod="vllm-0"}[5m])

GPU memory vs. request queue : When GPU memory hits the limit, vLLM starts queuing. This is the first sign you need to either reduce --gpu-memory-utilization or add another replica.

Time to first token : The metric users actually feel. If this goes above 2 seconds, something needs attention.

Power draw : Not for alerting, but useful for cost estimation. I can correlate power draw with request volume to estimate per-request energy cost.

What I See That I Couldn't See Before

Last week GPU utilization dropped to 20% while request latency spiked to 5 seconds. With DCGM Exporter alone, I would've been confused the GPU looks fine, why is it slow?

With the combined OTel pipeline, I could see that the batch scheduler in vLLM was waiting for the next batch window. The model had just been loaded (I'd updated the deployment), and the KV cache was cold. Throughput recovered in about 30 seconds as the cache warmed up. Without the application metrics alongside the GPU metrics, I would have been debugging this for an hour.

OCI APM Integration

OCI has its own APM service that accepts OpenTelemetry data. The nice thing about sending metrics there is that you get OCI-native alerting and integration with OCI notifications:

# Create an alarm in OCI Monitoring
oci monitoring alarm create \
  --compartment-id $COMPARTMENT_ID \
  --display-name "GPU inference latency high" \
  --metric-compartment-id $COMPARTMENT_ID \
  --namespace "custom_metrics" \
  --query 'vllm_e2e_request_latency_seconds[5m]{p99}.max() > 5' \
  --severity CRITICAL \
  --destinations '["'$NOTIFICATION_TOPIC_ID'"]'

This sends a PagerDuty/Slack/email alert when p99 inference latency exceeds 5 seconds. The GPU metrics and application metrics are in the same namespace, so you can write alarms that reference both.

My Dashboard Panels

If you're setting this up, here's what I'd put on the Grafana dashboard:

GPU Utilization + Tokens/sec - dual-axis, should correlate
GPU Memory Used / Total - with a threshold line at 90%
Request Queue Depth - should be near zero during normal ops
Time to First Token (p50, p95, p99) - the metric that matters to users
Pod Restart Count - OOM kills show up here
GPU Temperature - more for curiosity, but useful for thermal throttling detection

The shift from "GPU dashboard + separate app dashboard" to "one unified dashboard" made debugging 10x faster. OpenTelemetry is the glue that makes it work.

Pavan Madduri - Oracle ACE Associate, CNCF Golden Kubestronaut. Author of otel-gpu-receiver. GitHub | LinkedIn | Website | Google Scholar | ResearchGate

Docker Build Cloud Cut My CI Build Times by 75%, Here's How I Wired It to OCIR

Pavan Madduri — Wed, 01 Jul 2026 14:41:10 +0000

My GPU inference image was 8GB. Building it in GitHub Actions took 14 minutes on the free runner. Pushing to OCIR took another 6 minutes. Twenty minutes of CI time for every commit to main. I was burning through GitHub Actions minutes and my team was complaining about slow deployments.

Docker Build Cloud offloads the build to Docker's remote builders. They have fast machines with big caches. My 14-minute build dropped to 3 minutes. The push stayed at 6 minutes (that's network, can't speed it up much), but total CI went from 20 minutes to about 9.

What Docker Build Cloud Is

It's a remote build service from Docker. Instead of building on your CI runner (which is usually a small VM with cold layer caches), you build on Docker's infrastructure. They cache your layers aggressively across builds.

You don't change your Dockerfile. You add one setup step to your CI pipeline and change the docker build command to use the cloud builder. That's it.

The Before Pipeline

This is what I had:

# .github/workflows/build.yml (before)
name: Build and Push
on:
  push:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Login to OCIR
        run: echo "${{ secrets.OCIR_TOKEN }}" | docker login iad.ocir.io -u "${{ secrets.OCIR_USER }}" --password-stdin

      - name: Build image
        run: docker build -t iad.ocir.io/${{ secrets.TENANCY }}/inference/vllm:${{ github.sha }} .

      - name: Push to OCIR
        run: docker push iad.ocir.io/${{ secrets.TENANCY }}/inference/vllm:${{ github.sha }}

Build: 14 min. Push: 6 min. Total: ~20 min.

The GitHub Actions runner starts fresh every time — no layer cache. Every apt-get install, every pip install, every go mod download runs from scratch on every build.

The After Pipeline

# .github/workflows/build.yml (after — with Docker Build Cloud)
name: Build and Push
on:
  push:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
        with:
          driver: cloud
          endpoint: "pmady/oci-builds"     # your Build Cloud org/builder

      - name: Login to Docker (for Build Cloud auth)
        uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKERHUB_USER }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

      - name: Login to OCIR
        uses: docker/login-action@v3
        with:
          registry: iad.ocir.io
          username: ${{ secrets.OCIR_USER }}
          password: ${{ secrets.OCIR_TOKEN }}

      - name: Build and push
        uses: docker/build-push-action@v5
        with:
          push: true
          tags: |
            iad.ocir.io/${{ secrets.TENANCY }}/inference/vllm:${{ github.sha }}
            iad.ocir.io/${{ secrets.TENANCY }}/inference/vllm:latest
          cache-from: type=registry,ref=iad.ocir.io/${{ secrets.TENANCY }}/inference/vllm:cache
          cache-to: type=registry,ref=iad.ocir.io/${{ secrets.TENANCY }}/inference/vllm:cache,mode=max

Build: 3 min (layers cached on Build Cloud). Push: 6 min. Total: ~9 min.

The big difference is the driver: cloud in the Buildx setup. That sends the build to Docker's builders instead of running it on the GitHub runner. Their builders are faster machines and they persist your layer cache between builds.

Why the Build Is So Much Faster

Three reasons:

1. Persistent layer cache. On GitHub Actions, every build starts from nothing. On Build Cloud, my base image layers, dependency downloads, and compilation results are all cached from the previous build. Only the layers that actually changed get rebuilt.

2. Better hardware. Build Cloud builders have more CPU, more RAM, and faster disks than the free GitHub Actions runners. The compilation step alone runs 2-3x faster.

3. Parallel multi-platform builds. When I build multi-arch images (amd64 + arm64), Build Cloud runs both architectures in parallel on native hardware. No QEMU emulation. On GitHub Actions, the ARM build was emulated and took 4x longer.

Registry Cache on OCIR

I store the build cache in OCIR itself using the cache-to: type=registry option. This means:

Build Cloud uses its own fast cache for immediate rebuilds
The registry cache in OCIR acts as a secondary cache that any CI runner can pull from
No external cache service needed

The cache image (vllm:cache) is separate from the production image. It contains all the intermediate layers and gets updated on every build. It's large (~3GB) but OCIR storage is cheap.

Multi-Arch With Build Cloud

This is where Build Cloud really shines. Building ARM images on GitHub Actions with QEMU was painfully slow:

# With Build Cloud — native ARM + AMD64 builds in parallel
- name: Build and push multi-arch
  uses: docker/build-push-action@v5
  with:
    push: true
    platforms: linux/amd64,linux/arm64
    tags: iad.ocir.io/${{ secrets.TENANCY }}/myapp:${{ github.sha }}

Build Type	GitHub Actions (QEMU)	Docker Build Cloud
AMD64 only	14 min	3 min
ARM64 only (emulated)	45 min	4 min
Multi-arch (both)	50 min	5 min

The multi-arch build went from 50 minutes to 5 minutes. That's not an optimization, that's a category change.

Cost

Docker Build Cloud has a free tier (200 build minutes/month) and paid plans. For my usage (~50 builds/month averaging 3 minutes each = 150 minutes), the free tier covers it.

Compare that to what I was spending on GitHub Actions: 20 min × 50 builds = 1,000 minutes/month. The free tier has 2,000 minutes, so I wasn't paying, but I was burning through them fast. Now I use 150 minutes of Build Cloud (free) and ~300 minutes of GitHub Actions (for the push + deploy steps).

The Setup Cost

Getting Build Cloud connected to OCIR took about 15 minutes:

Create a Docker Build Cloud builder in Docker Hub settings
Add DOCKERHUB_USER and DOCKERHUB_TOKEN to GitHub secrets
Change the docker/setup-buildx-action driver to cloud
Add the OCIR registry cache config

No infrastructure to manage, no self-hosted runners to maintain, no cache servers to run.

Pavan Madduri — Oracle ACE Associate, CNCF Golden Kubestronaut. GitHub | LinkedIn | Website | Google Scholar | ResearchGate

How I Run GPU Workloads for 70% Less on OKE Using Preemptible Instances

Pavan Madduri — Tue, 30 Jun 2026 15:35:49 +0000

I was spending ~$3,300/month on three A10 GPU instances for a mix of staging inference, batch processing, and experimentation. All on-demand. Then I switched two of the three to preemptible instances and my GPU bill dropped to about $1,450/month.

The trade-off is that OCI can reclaim preemptible instances with 30 seconds notice. For customer-facing production inference, that's a non-starter. For everything else I was running, it was fine.

What Preemptible Instances Actually Mean

OCI preemptible instances are spare capacity sold at a steep discount. A10 GPU goes from $1.52/hr (on-demand) to ~$0.46/hr (preemptible). That's a 70% discount.

The catch: OCI can terminate the instance when it needs the capacity back. You get a 30-second warning via instance metadata. Your workload needs to handle this gracefully.

In practice, I've been running preemptible GPU nodes on OKE for two months and I've had maybe 4-5 evictions total. Some weeks zero. It depends on demand in your region and availability domain.

OKE Node Pool Setup

I run two GPU node pools one on-demand for production, one preemptible for everything else:

# Production GPU pool always available
oci ce node-pool create \
  --name gpu-production \
  --node-shape VM.GPU.A10.1 \
  --node-config-details '{
    "size": 1,
    "placementConfigs": [{
      "availabilityDomain": "Uocm:US-ASHBURN-AD-1",
      "subnetId": "'$SUBNET_ID'",
      "preemptibleNodeConfig": null
    }]
  }' \
  ...

# Preemptible GPU pool cheap, may get evicted
oci ce node-pool create \
  --name gpu-preemptible \
  --node-shape VM.GPU.A10.1 \
  --node-config-details '{
    "size": 2,
    "placementConfigs": [{
      "availabilityDomain": "Uocm:US-ASHBURN-AD-1",
      "subnetId": "'$SUBNET_ID'",
      "preemptibleNodeConfig": {
        "preemptionAction": {
          "type": "TERMINATE",
          "isPreserveBootVolume": false
        }
      }
    }]
  }' \
  --node-metadata '{"user_data": "..."}' \
  --initial-node-labels '[
    {"key": "node-type", "value": "preemptible"},
    {"key": "nvidia.com/gpu", "value": "present"}
  ]'

The label node-type=preemptible is how I control which workloads land on preemptible nodes.

Directing Workloads to the Right Pool

Production inference uses a node affinity to avoid preemptible nodes:

# Production on-demand only
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-production
spec:
  template:
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: node-type
                    operator: NotIn
                    values: ["preemptible"]
      containers:
        - name: vllm
          resources:
            limits:
              nvidia.com/gpu: 1

Staging and batch workloads prefer preemptible (but tolerate on-demand if preemptible nodes are unavailable):

# Staging prefer preemptible, accept on-demand as fallback
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-staging
spec:
  template:
    spec:
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 90
              preference:
                matchExpressions:
                  - key: node-type
                    operator: In
                    values: ["preemptible"]
      tolerations:
        - key: "preemptible"
          operator: "Exists"
          effect: "NoSchedule"
      containers:
        - name: vllm
          resources:
            limits:
              nvidia.com/gpu: 1

Handling Eviction Gracefully

When OCI reclaims a preemptible instance, the node drains and pods get terminated. For inference services, this means requests in flight get dropped. Here's how I handle it:

1. Pod Disruption Budget

Prevents all replicas from being evicted simultaneously (only matters if you have >1 replica):

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: vllm-staging-pdb
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: vllm-staging

2. Graceful Shutdown

vLLM handles SIGTERM and finishes in-flight requests before shutting down. I set terminationGracePeriodSeconds to 25 (less than the 30-second eviction notice):

spec:
  terminationGracePeriodSeconds: 25
  containers:
    - name: vllm
      lifecycle:
        preStop:
          exec:
            command: ["/bin/sh", "-c", "sleep 5"]

The 5-second preStop sleep gives the load balancer time to stop sending new requests before the container starts shutting down.

3. Model Cache on PVC

When a pod gets evicted and rescheduled to a new preemptible node, I don't want to re-download the model from scratch. I use the OCI Object Storage init container approach (from my earlier post) so the model loads in ~90 seconds instead of 12 minutes.

The Actual Numbers

Two months of data:

Metric	Value
Eviction events	9 total (avg ~1/week)
Avg time to recover	~2 minutes (pod reschedule + model load)
Longest outage	4 minutes (node provisioning + model load)
Monthly cost (3x on-demand)	$3,282
Monthly cost (1x on-demand + 2x preemptible)	~$1,450
Savings	$1,832/month (56%)

The evictions cluster — I had three in one day during what I assume was a capacity crunch in us-ashburn-1, then nothing for two weeks. Unpredictable, but the recovery is fast enough that nobody on the team complained.

Batch Jobs on Preemptible Even Better

For batch inference (processing a dataset, not serving live traffic), preemptible is almost a no-brainer. I use Kubernetes Jobs with checkpointing:

apiVersion: batch/v1
kind: Job
metadata:
  name: batch-inference
spec:
  backoffLimit: 5    # retry up to 5 times if evicted
  template:
    spec:
      nodeSelector:
        node-type: preemptible
      restartPolicy: OnFailure
      containers:
        - name: inference
          image: iad.ocir.io/mytenancy/batch-inference:v1
          env:
            - name: CHECKPOINT_BUCKET
              value: "inference-checkpoints"
          resources:
            limits:
              nvidia.com/gpu: 1

The batch job saves progress to OCI Object Storage every N records. If it gets evicted, Kubernetes restarts it and it picks up from the last checkpoint. I've had batch jobs complete across 3 evictions without losing any work.

When to Use Preemptible GPUs

Staging/dev environments: latency spikes from eviction are fine
Batch inference: checkpoint and retry
Training runs: if your framework supports checkpointing (most do)
Experimentation: exploring models, testing prompts

When Not To

Customer-facing inference: use on-demand, the cost is worth the reliability
Short-deadline batch: if the job must finish by a specific time, eviction adds unpredictability
Single-replica production: no fallback when the one instance gets evicted

Pavan Madduri: Oracle ACE Associate, CNCF Golden Kubestronaut. GitHub | LinkedIn | Website | Google Scholar | ResearchGate

Running Ollama on OCI Container Instances - Private LLM API in 5 Minutes, No Kubernetes

Pavan Madduri — Mon, 29 Jun 2026 14:53:58 +0000

A colleague asked me to set up a private LLM endpoint their team could use for code review suggestions. Requirements: OpenAI-compatible API, runs inside our cloud (no data leaving the tenancy), and "I don't want to learn Kubernetes."

That last requirement ruled out OKE. And honestly, for a single-model inference endpoint serving 10 people, Kubernetes is overkill anyway.

I had Ollama running on an OCI Container Instance with a GPU in about 5 minutes. Here's the whole thing.

Why Ollama Instead of vLLM

For a small team endpoint, Ollama wins on simplicity:

Single binary, no Python dependencies
Downloads models automatically on first run
Manages multiple models with simple ollama pull
Built-in OpenAI-compatible API at /v1/chat/completions
Handles model loading/unloading from GPU memory automatically

vLLM is better for high-throughput production (continuous batching, PagedAttention), but this isn't that. This is "10 developers hitting it a few times an hour."

The Deployment

One CLI command:

oci container-instances container-instance create \
  --compartment-id $COMPARTMENT_ID \
  --availability-domain "Uocm:US-ASHBURN-AD-1" \
  --display-name "team-ollama" \
  --shape "CI.Standard.GPU.A10.1" \
  --shape-config '{"ocpus": 15, "memoryInGBs": 240}' \
  --containers '[{
    "imageUrl": "docker.io/ollama/ollama:latest",
    "displayName": "ollama",
    "resourceConfig": {
      "vcpusLimit": 15,
      "memoryLimitInGBs": 240
    },
    "environmentVariables": {
      "OLLAMA_HOST": "0.0.0.0"
    },
    "healthChecks": [{
      "healthCheckType": "HTTP",
      "port": 11434,
      "path": "/",
      "intervalInSeconds": 30
    }]
  }]' \
  --vnics '[{
    "subnetId": "'$PRIVATE_SUBNET_ID'",
    "isPublicIpAssigned": false
  }]'

Few things to note:

GPU shape — CI.Standard.GPU.A10.1 gives you an A10 GPU with 24GB VRAM. Enough for most 7-13B parameter models.
Private subnet — No public IP. The endpoint is only accessible from within the VCN. I added a bastion or VPN for the team to reach it.
OLLAMA_HOST=0.0.0.0 — By default Ollama only listens on localhost. Inside a container, you need it to listen on all interfaces.

The container starts in about 10 seconds. But no model is loaded yet.

Loading the Model

Ollama downloads models on first use. I SSH'd through the bastion and triggered the first pull:

# From a VM in the same VCN
OLLAMA_IP=10.0.1.42  # private IP of the Container Instance

# Pull a model (downloads to container's filesystem)
curl http://$OLLAMA_IP:11434/api/pull -d '{"name": "llama3.1:8b"}'

# Test it
curl http://$OLLAMA_IP:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.1:8b",
    "messages": [{"role": "user", "content": "Review this Go function for bugs: func add(a, b int) int { return a - b }"}]
  }'

The initial model download takes 3-4 minutes (7B model, ~4GB). After that, responses start within a second or two.

The Problem: Model Persistence

Here's the catch I should have thought about earlier. Container Instances don't have persistent storage by default. If the container restarts, the downloaded model is gone. You have to pull it again.

My fix was mounting an OCI Block Volume:

oci container-instances container-instance create \
  ... \
  --containers '[{
    "imageUrl": "docker.io/ollama/ollama:latest",
    "volumeMounts": [{
      "mountPath": "/root/.ollama",
      "volumeName": "model-storage"
    }]
  }]' \
  --volumes '[{
    "name": "model-storage",
    "volumeType": "EMPTYDIR",
    "backingStore": "EPHEMERAL_STORAGE"
  }]'

For true persistence across container recreations, you'd use an OCI File Storage (NFS) mount. But for this use case, the ephemeral storage survives restarts (not recreations), and I have a simple curl script that re-pulls the model if it's missing:

#!/bin/bash
# warmup.sh — run after container instance creation
OLLAMA_IP=$1

# Wait for Ollama to be ready
until curl -sf http://$OLLAMA_IP:11434/ > /dev/null; do
  sleep 2
done

# Pull model if not present
curl -sf http://$OLLAMA_IP:11434/api/show -d '{"name":"llama3.1:8b"}' > /dev/null 2>&1
if [ $? -ne 0 ]; then
  echo "Pulling model..."
  curl http://$OLLAMA_IP:11434/api/pull -d '{"name": "llama3.1:8b"}'
fi

echo "Ready at http://$OLLAMA_IP:11434"

What the Team Uses It For

The endpoint has been running for three weeks. The team uses it for:

Code review suggestions — paste a function, ask for review
Commit message generation — describe changes, get a conventional commit
Documentation drafts — generate docstrings and README sections
SQL query help — describe what they want, get a query back

Traffic is light — maybe 50-100 requests/day total. The A10 GPU sits at 5-15% utilization most of the time. It's overkill, but even overkill on OCI is only ~$1,094/month, and the team finds it useful enough to justify the cost.

Cost vs. Alternatives

Option	Monthly Cost	Setup Time
OCI Container Instance + A10 GPU	~$1,094	5 minutes
OpenAI API (estimated 100 req/day)	~$30-150	N/A
Self-hosted on OKE	~$1,094 + complexity	30-60 minutes

Yeah, OpenAI is cheaper for this volume. But the team's requirement was "no data leaving our cloud." Compliance rules. The Container Instance approach gave them a private endpoint with zero Kubernetes complexity. Sometimes you pay for simplicity and privacy.

If I Were Doing This Again

I'd use OCI File Storage instead of ephemeral storage so models survive container recreation. And I'd put an OCI API Gateway in front of it for rate limiting and auth, instead of relying on network-level access control. The gateway adds about $50/month but gives you proper API keys and request logging.

For teams larger than ~20 people or with higher throughput needs, I'd switch to vLLM on OKE with the setup I described in my earlier posts. But for a small team that just wants a private LLM without touching Kubernetes? Ollama on Container Instances is hard to beat for simplicity.

Pavan Madduri — Oracle ACE Associate, CNCF Golden Kubestronaut. GitHub | LinkedIn | Website | Google Scholar | ResearchGate

Zero-Downtime Crossplane v1 v2 Migration: Adopt-in-Place at Production Scale

Pavan Madduri — Fri, 26 Jun 2026 01:33:13 +0000

Crossplane v2 (released in late 2025) introduced a cleaner, namespaced resource model and removed a lot of the v1 ceremony around Claims and cluster-scoped composites. Upgrading the control plane to v2 is usually painless — if you're not using the v1 features that changed, your existing claims keep working thanks to backward compatibility.

The hard part is the next step: migrating your existing v1-style workloads onto v2-style namespaced resources. That's where there's still no cohesive, end-to-end story — and it's where I spent most of my effort taking a production EKS fleet all the way through.

This post is the field guide I wish I'd had: the adopt-in-place method, how to validate it before touching anything, and the three failure modes that will bite you in production.

Nothing here destroys or recreates cloud infrastructure. The whole point is to keep every existing AWS resource exactly where it is and just change which Crossplane resource owns it.

The setup (in generic terms)

A control plane running Crossplane, managing Amazon EKS clusters end to end.
Each cluster is represented by a Crossplane composite, which in turn owns ~90–100 managed resources (MRs): IAM roles/policies, the EKS cluster, EKS add-ons, a managed NodeGroup, a launch template, security groups and rules, an OIDC provider, and a pile of Objects managed through provider-kubernetes.
Several distinct cluster archetypes, each backed by its own Composition (think: general workload clusters, ingress/gateway clusters, and stateful clusters). Same migration mechanics, slightly different resource sets.
GitOps-driven: a Git repository is the source of truth, reconciled by a GitOps controller.

Constraints that shaped everything: no resource recreation, no node rotation, zero downtime.

Why "delete and recreate" is a non-starter

The naive migration is: delete the v1 composite, create the v2 XR, let the provider rebuild everything. In production that's a non-starter:

You cannot destroy a VPC, an EKS control plane, or a live NodeGroup and rebuild it under traffic.
Even Crossplane's Observe /import flows leave a window where the resource is briefly unmanaged or re-created.
Anything that recreates a NodeGroup triggers a node rotation — every pod on the cluster gets evicted and rescheduled. That's a customer-visible event you do not want as a side effect of an internal refactor.

So the goal isn't "create v2 resources." It's "make a v2 XR adopt the exact resources the v1 composite already owns, with zero observable change."

How Crossplane decides what to create vs. adopt

Two facts make adopt-in-place possible:

External-name is the source of truth for the real cloud resource. Crossplane reconciles a managed resource against the actual AWS object identified by its crossplane.io/external-name annotation. If a v2-owned MR has the same external-name as the live AWS resource, Crossplane observes it instead of creating a new one.
Ownership is expressed by a label + an ownerReference. A composed MR points back to its owning composite via:
- the crossplane.io/composite label, and
- a Kubernetes ownerReference to the owning XR (name + UID).
Within a composition, each MR is keyed by a "composition-resource-name" (crn) — the crossplane.io/composition-resource-name annotation. The engine matches the desired resource the composition wants to produce against the observed MR with the same crn. Same crn + same external-name → adopt in place. Different crn → the engine thinks the desired resource is missing and creates a new one (and treats the old one as an orphan).

Adopt-in-place is just: rewrite ownership (#2) and make crn + external-name line up (#1, #3) so the v2 composition's desired output matches what's already there.

The adopt-in-place method, step by step

Step 0 — Snapshot and pre-validate (do this before touching prod)

Before any mutation, capture the live state and prove the v2 composition will adopt rather than recreate. Crossplane's render command can do this offline against observed resources:

crossplane beta render \
  xr.yaml \
  composition.yaml \
  functions.yaml \
  --observed-resources ./observed/

./observed/ holds the live MRs (exported from the cluster). The command prints the desired resources the v2 composition would produce. Diff desired vs. observed and classify every resource:

adopt-in-place — desired crn (after remap, see below) and external-name match an observed MR.
net-new — desired resource the v2 composition adds that v1 didn't have.
orphan — observed MR that the v2 composition doesn't produce.

Gate the migration on: zero orphans, zero unexpected net-new. This single offline check caught every surprise before it reached production.

Step 1 — Pause both the claim and the composite

kubectl annotate <claim-kind> <name> crossplane.io/paused="true"
kubectl annotate <composite-kind> <name> crossplane.io/paused="true"

Pausing only the claim is a classic mistake — the composite keeps reconciling and will fight you. Pause both.

Step 2 — Reparent every managed resource

For each MR owned by the v1 composite, repoint ownership at the new v2 XR:

set the crossplane.io/composite label to the v2 XR name,
replace the ownerReference with one pointing at the v2 XR (kind, name, UID).

Conceptually:

metadata:
  labels:
    crossplane.io/composite: <v2-xr-name>     # was: <v1-composite-name>
  ownerReferences:
    - apiVersion: <v2-xr-apiVersion>
      kind: <v2-xr-kind>
      name: <v2-xr-name>
      uid: <v2-xr-uid>                          # the new XR's uid
      controller: true
      blockOwnerDeletion: true

Script this across all ~90–100 MRs; doing it by hand is how you get an inconsistent state.

Step 3 — Point the v2 XR at the adopted resources

Patch the v2 XR so it references the composition and the exact resources it's adopting:

spec:
  crossplane:
    compositionRef:
      name: <v2-composition>
    resourceRefs:
      - { apiVersion: ..., kind: ..., name: <mr-1> }
      - { apiVersion: ..., kind: ..., name: <mr-2> }
      # ... all adopted MRs

Keep the XR paused while you do this (create it paused from the start).

Step 4 — Unpause and let it converge

kubectl annotate <v2-xr-kind> <name> crossplane.io/paused-

The engine reconciles, matches desired↔observed by crn + external-name, and adopts. Watch the XR and its MRs go Synced=True/Ready=True without any Create calls hitting AWS.

The three failure modes that will bite you

1. NodeGroup composition-resource-name drift (blue/green)

This is the one most likely to cause a real incident.

Our v1 composition emitted the managed NodeGroup with one crn (e.g. a blue/green-style nodegroup-active), while the v2 composition emits a different crn (e.g. nodegroup). Because the engine matches desired↔observed by crn, the mismatch means:

the v2 composition's desired nodegroup has no matching observed MR → it wants to create one, and
the live NodeGroup (crn nodegroup-active) has no matching desired resource → it's treated as an orphan.

The net effect would be a brand-new NodeGroup and a rotation of every node.

Fix: remap the crn annotation on the live NodeGroup to match what the v2 composition expects, and preserve the existing NodeGroup name/external-name. Don't touch external-name — that's what keeps it bound to the real AWS NodeGroup.

kubectl annotate nodegroup.<group> <existing-ng-name> \
  crossplane.io/composition-resource-name=nodegroup --overwrite

Verify no rotation after cutover by confirming the launch-template name+version and the NodeGroup version are unchanged from before:

kubectl get nodegroup.<group> <name> \
  -o jsonpath='LT={.status.atProvider.launchTemplate.name}:v{.status.atProvider.launchTemplate.version} ver={.status.atProvider.version}'

Same values before and after = the existing nodes were adopted, not replaced.

2. The cluster-auth connection-secret republish race

The subtlest one — a silent failure if you're not watching for it.

For EKS, a managed "cluster auth" resource generates the kubeconfig (a short-lived token) and writes it to a connection Secret. The provider-kubernetes ProviderConfig reads that Secret to talk to the workload cluster, and every Object on that cluster depends on it.

When the v2 XR took ownership, the connection Secret got recreated empty. If the cluster-auth resource's last token refresh happened before that recreation, it didn't immediately republish — so the Secret stayed empty. Every downstream Object then stranded with:

cannot build kube client for provider config: currentContext not set in kubeconfig

On most clusters this self-healed on the cluster-auth resource's next refresh cycle. On one, the timing left it stuck for several minutes with no sign of recovering.

Fix: force the cluster-auth resource to reconcile so it republishes the kubeconfig. A benign annotation bump does it:

kubectl annotate <clusterauth-kind> <name> \
  example.com/republish="$(date -u +%s)" --overwrite

The connection Secret repopulates, and the stranded Objects build their client and sync. The lesson: adopting an MR can re-create its connection Secret out from under downstream consumers. Put health checks on the downstream objects, not just the cluster resource, or you'll never see it.

3. GitOps source-of-truth drift

The live cutover above is imperative. Your GitOps repo still describes the v1 world. Until you reconcile it, your GitOps controller will try to "fix" the cluster back toward the manifests — unpausing the v1 claim, or having no record of the v2 XR at all.

Treat the cluster migration and the source-of-truth migration as two separate workstreams. After the live cutover, land a Git change that:

adds crossplane.io/paused: "true" to the v1 claim manifests, and
adds the v2 XR manifests without the paused annotation (so the controller manages them as the active resources).

Make sure auto-sync/self-heal won't revert your live state in the gap between cutover and merge.

A repeatable runbook

Boiled down, every cluster followed the same pipeline:

snapshot — export live MRs, claim, composite, and composition.
render-gate — beta render --observed-resources + diff; require zero orphans / zero unexpected net-new; confirm the NodeGroup crn remap and that launch-template/version match.
reparent — script the label + ownerReference rewrite for all MRs (with a rollback script that puts them back).
patch — set the v2 XR compositionRef + resourceRefs (still paused).
pause v1 → unpause v2.
health-gate — every MR Synced/Ready excluding a known-baseline set; NodeGroup unchanged; downstream Objects connected.
reconcile Git — pause v1 manifests, add unpaused v2 manifests.

Do non-prod first, build the runbook, then prod. Keep the v1 composite paused (not deleted) for a cooldown period so rollback is a single unpause away.

What the ecosystem still needs

Most of the above was hand-rolled. A few things would turn this from "expert-only surgery" into a supported workflow:

A migrate command that, given a v1 claim/composite and a target v2 composition, generates the reparent patches, the v2 XR with populated resourceRefs, and — critically — a crn remap table between the two compositions. Matching must be by external resource identity, not crn string equality.
An adopt-preview/dry-run that classifies every MR as adopt-in-place / net-new / orphan and gates on zero orphans before proceeding (productizing the render --observed-resources diff).
Connection-secret-aware adoption — on adoption, force a reconcile or wait on connection-secret readiness so downstream providers don't lose connectivity.

There's an active community effort around exactly this (a maintainer-run feedback discussion and a migration-tooling tracking issue, plus a community CLI for migrating composition manifests). If you've done a migration like this, your war stories are genuinely useful input — the design is still being shaped.

Takeaways

Adopt, don't recreate. Make a v2 XR own the exact MRs the v1 composite owned; never let external-names change.
Validate offline first. beta render --observed-resources + a desired/observed diff is the single highest-leverage safety check.
crn alignment is everything for NodeGroups — a mismatch is the difference between a silent adoption and a full node rotation.
Watch your connection secrets. Adoption can recreate them empty; downstream consumers fail silently until the owner republishes.
Two migrations, not one. The live cluster and the GitOps source of truth move separately — plan for both.

It's very doable to take a production fleet from v1 to v2 with zero downtime today — it just isn't yet a one-command experience. Hopefully this shortens the path for the next person.

Give Your AI Agent Eyes on GPUs: Introducing gpu-mcp-server

Pavan Madduri — Wed, 24 Jun 2026 19:34:39 +0000

Your AI agent can write code, search the web, and query databases. But ask it what's happening on your GPUs right now utilization, memory pressure, thermals and it draws a blank. That's because there's been no standard way for agents to access hardware telemetry.

gpu-mcp-server changes that. It's an open source MCP server that gives any AI agent real-time access to NVIDIA GPU metrics through the Model Context Protocol.

The Problem

If you're running GPU workloads inference servers, training jobs, fine-tuning pipelines you probably check GPU status the same way everyone does:

nvidia-smi

That works for humans. It doesn't work for agents.

When an agent needs to decide whether to scale up replicas, diagnose a slow inference endpoint, or figure out why a training job OOM'd, it has no way to see the hardware layer. It's flying blind.

Monitoring stacks like Prometheus + dcgm-exporter solve this for dashboards, but they're heavy infrastructure that agents can't easily query through MCP.

What gpu-mcp-server Does

It's a single Go binary that runs alongside your agent and exposes three tools over the Model Context Protocol:

Tool	What It Does
`list_gpus`	Lists every GPU on the machine with utilization and memory
`get_gpu_metrics`	Deep dive on one GPU temp, power, PCIe/NVLink throughput
`gpu_summary`	Aggregate stats across all devices

The server talks to GPUs through NVIDIA's NVML C library directly - no Prometheus, no metric pipelines, no extra infrastructure.

MIG Support Built In

If you're running Multi-Instance GPU (MIG) on A100s or H100s, each MIG instance shows up as a separate device. Shared metrics (temperature, power, PCIe) are pulled from the physical GPU. Per-instance metrics (utilization, memory) come from the MIG slice.

Why MCP?

The Model Context Protocol is an open standard now hosted by the Agentic AI Foundation at the Linux Foundation for connecting AI agents to tools and data sources. Every major AI platform supports it: Claude, Goose, Cursor, Windsurf, and more.

By implementing GPU metrics as an MCP server, any of these agents can access your GPU state without custom integrations.

Setup

Build

git clone https://github.com/pmady/gpu-mcp-server.git
cd gpu-mcp-server
make build   # requires CGO + NVIDIA drivers

Claude Desktop

Add to claude_desktop_config.json:

{
  "mcpServers": {
    "gpu": {
      "command": "/path/to/gpu-mcp-server"
    }
  }
}

Goose

extensions:
  gpu-metrics:
    type: stdio
    cmd: /path/to/gpu-mcp-server

Once connected, you can ask your agent things like:

"Which GPU has the most memory available?"
"Are any GPUs thermal throttling right now?"
"What's the total power draw across all devices?"
"Show me the MIG instances on GPU 0"

The agent calls the right tool, gets structured JSON back, and reasons over it.

Architecture

Agent (Claude / Goose / Cursor)
          │
          │ MCP (stdio)
          ▼
    gpu-mcp-server
          │
          │ NVML (cgo)
          ▼
    NVIDIA GPU Hardware

No sidecar. No network hops. No metric pipeline. The server runs as a local process and calls NVML directly through cgo. If you can run nvidia-smi, you can run this.

Who Is This For?

ML engineers who want agents to diagnose GPU workload issues
Platform teams building agentic infrastructure on GPU clusters
Anyone running inference servers (vLLM, Triton, SGLang) who wants agents to understand hardware utilization

Relationship to keda-gpu-scaler

If you've seen keda-gpu-scaler - my KEDA external scaler for GPU-based autoscaling — this shares the same NVML foundation. Where keda-gpu-scaler uses GPU metrics to drive Kubernetes autoscaling decisions, gpu-mcp-server exposes those same metrics to AI agents through MCP.

Same GPU telemetry engine, two different consumers: orchestrators and agents.

What's Next

Streamable HTTP transport - run as a network service, not just stdio
Resource endpoints - expose GPU info as MCP resources for agent context
Process-level GPU usage - per-PID memory and compute attribution
AAIF project proposal - working toward official AAIF hosted project status

Get Involved

The project is Apache 2.0 and contributions are welcome:

GitHub: github.com/pmady/gpu-mcp-server
Issues: Bug reports and feature requests
PRs: See CONTRIBUTING.md

If you're running GPU workloads and using MCP-compatible agents, give it a try.
Star the repo if it's useful. Open an issue if it's not.

Pavan Madduri is a platform engineer focused on GPU-aware AI infrastructure.
He maintains keda-gpu-scaler and
contributes to CNCF projects including Dragonfly, Volcano, and KEDA.

docker init OCIR OKE: From Empty Folder to Production in 15 Minutes

Pavan Madduri — Wed, 24 Jun 2026 01:36:06 +0000

I timed myself. Starting from an empty directory with a Go application idea, how fast could I get to a running deployment on OKE? The answer was 14 minutes. docker init did more of the work than I expected.

What docker init Does

If you haven't used it, docker init is an interactive scaffolding tool built into Docker CLI. You run it in your project directory and it generates a Dockerfile, .dockerignore, and docker-compose.yml tuned for your language.

mkdir oci-api && cd oci-api
go mod init github.com/pmady/oci-api

# Write a quick API
cat > main.go << 'EOF'
package main

import (
    "fmt"
    "log"
    "net/http"
    "os"
)

func main() {
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        fmt.Fprintf(w, "running on %s", os.Getenv("OCI_REGION"))
    })
    http.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) {
        w.WriteHeader(200)
    })
    log.Fatal(http.ListenAndServe(":8080", nil))
}
EOF

Now the magic part:

$ docker init

Welcome to the Docker Init CLI!

? What application platform does your project use? Go
? What version of Go do you want to use? 1.22
? What's the relative directory for your main package? .
? What port does your server listen on? 8080

It generates three files:

Dockerfile — Multi-stage build, distroless base, non-root user. Actually good defaults. I've seen teams write worse Dockerfiles by hand.

compose.yaml — Basic setup with port mapping and env vars.

.dockerignore — Excludes .git, binaries, vendor directory. Reasonable.

The generated Dockerfile looked like this (slightly simplified):

FROM golang:1.22 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o /bin/server .

FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /bin/server /bin/
EXPOSE 8080
USER nonroot:nonroot
ENTRYPOINT ["/bin/server"]

Multi-stage, static binary, distroless, non-root. I'd write almost the same thing myself. The only change I made was adding -ldflags="-s -w" to strip debug symbols and shrink the binary.

Minute 0-5: Build and Test Locally

docker compose up --build

# In another terminal
curl localhost:8080
# running on

curl localhost:8080/health
# 200 OK

Works. Five minutes in and I have a containerized API running locally.

Minute 5-8: Push to OCIR

# Login
docker login iad.ocir.io -u '<tenancy-namespace>/pmady'

# Tag
docker tag oci-api-server:latest iad.ocir.io/<tenancy>/demos/oci-api:v1

# Quick scan
docker scout cves iad.ocir.io/<tenancy>/demos/oci-api:v1

# Push
docker push iad.ocir.io/<tenancy>/demos/oci-api:v1

Scout showed zero CVEs because distroless has almost nothing in it. Push took about 10 seconds because the image is 12MB.

Minute 8-14: Deploy to OKE

I already had an OKE cluster running (if you don't, add 20 minutes for oci ce cluster create). The deployment manifest:

# deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: oci-api
spec:
  replicas: 2
  selector:
    matchLabels:
      app: oci-api
  template:
    metadata:
      labels:
        app: oci-api
    spec:
      containers:
        - name: api
          image: iad.ocir.io/<tenancy>/demos/oci-api:v1
          ports:
            - containerPort: 8080
          env:
            - name: OCI_REGION
              value: us-ashburn-1
          readinessProbe:
            httpGet:
              path: /health
              port: 8080
            periodSeconds: 5
          resources:
            requests:
              cpu: 100m
              memory: 64Mi
            limits:
              cpu: 500m
              memory: 128Mi
      imagePullSecrets:
        - name: ocir-secret
---
apiVersion: v1
kind: Service
metadata:
  name: oci-api
  annotations:
    oci.oraclecloud.com/load-balancer-type: "lb"
    service.beta.kubernetes.io/oci-load-balancer-shape: "flexible"
    service.beta.kubernetes.io/oci-load-balancer-shape-flex-min: "10"
    service.beta.kubernetes.io/oci-load-balancer-shape-flex-max: "100"
spec:
  type: LoadBalancer
  selector:
    app: oci-api
  ports:
    - port: 80
      targetPort: 8080

kubectl apply -f deploy.yaml

# Wait for LB IP
kubectl get svc oci-api -w
# NAME      TYPE           CLUSTER-IP    EXTERNAL-IP      PORT(S)
# oci-api   LoadBalancer   10.96.1.120   129.153.xx.xx    80:31234/TCP

curl http://129.153.xx.xx/
# running on us-ashburn-1

14 minutes. Empty folder to public API on OKE.

What docker init Got Right

I was skeptical about docker init being useful for anything beyond demos. But the generated Dockerfile was genuinely good:

Multi-stage build — keeps the final image small
Distroless base — minimal attack surface, near-zero CVEs
Non-root user — security best practice out of the box
Separate dependency download — go mod download before COPY . means dependencies are cached and rebuilds are fast

The only things I changed for OKE deployment were the -ldflags optimization and adding a health check endpoint (which docker init can't know about since it's application-specific).

What It Doesn't Do

docker init handles the Docker side. It doesn't generate:

Kubernetes manifests
CI/CD pipeline config
OCIR login/push scripts
Terraform for infrastructure

That's fair. It's a Docker tool, not a platform tool. But the Dockerfile it generates is solid enough that I don't need to edit it for most Go and Python projects.

Languages I've Tested

Language	Quality of Generated Dockerfile	Notes
Go	Excellent	Multi-stage, static binary, distroless
Python	Good	Uses slim base, proper requirements.txt handling
Node.js	Good	Multi-stage, npm ci for production
Rust	Excellent	cargo-chef for caching, musl for static binary
Java	Decent	Uses Eclipse Temurin, could use jlink for smaller images

Go and Rust output is good enough to use as-is. Python and Node need minor tweaks depending on your framework. Java needs the most work.

My Workflow Now

For quick services and prototypes, this is my default:

mkdir project && cd project
# write code
docker init
# tweak Dockerfile if needed
docker compose up --build      # test locally
docker push ...                # push to OCIR
kubectl apply -f deploy.yaml   # deploy to OKE

The gap between "it works on my laptop" and "it's running on OKE" is smaller than it's ever been.

Pavan Madduri — Oracle ACE Associate, CNCF Golden Kubestronaut. GitHub | LinkedIn | Website | Google Scholar | ResearchGate

I Stopped Paying for Idle GPUs - Scale-to-Zero AI Inference on OKE with KEDA

Pavan Madduri — Wed, 17 Jun 2026 14:37:11 +0000

A single A10 GPU on OCI costs $1.52/hr. Running 24/7, that's $1,094/month. For a production inference service with steady traffic, that's fine. But I had a staging environment and a couple of internal tools that got maybe 20 requests per day. I was paying over $2,000/month for GPUs that sat idle 95% of the time.

The obvious solution: scale to zero when there's no traffic, spin up when a request comes in. KEDA does this on Kubernetes, but getting it to work properly with GPU pods took some figuring out.

Why Scaling GPUs Is Harder Than Scaling CPU Pods

With normal HTTP services, KEDA watches a metric (HTTP requests, queue depth, whatever), and Kubernetes can spin up a new pod in seconds. The user barely notices.

GPU pods are different:

Cold start is slow - Model loading takes 60-120 seconds
Node scaling is slow - If there's no GPU node in the pool, OKE needs to provision a new VM (3-5 minutes)
Images are huge - 5-15GB pull times if not cached
GPU resources are binary - You can't give a pod "half a GPU" (without MIG)

So you can't just scale-to-zero and expect sub-second response times when traffic returns. The trade-off is cost savings vs. cold start latency. For my use case (internal tools, staging), a 2-3 minute cold start was acceptable.

The Setup

1. Install KEDA on OKE

helm repo add kedacore https://kedacore.github.io/charts
helm install keda kedacore/keda \
  --namespace keda-system \
  --create-namespace

2. Prometheus for Request Metrics

I'm using the nginx ingress controller's Prometheus metrics to track request rate. If you're using OCI's native load balancer, you'd use OCI Monitoring metrics instead.

# prometheus-scaledobject.yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-scaler
  namespace: inference
spec:
  scaleTargetRef:
    name: vllm-inference
  minReplicaCount: 0          # scale to zero
  maxReplicaCount: 3
  cooldownPeriod: 300          # wait 5 min of no traffic before scaling down
  pollingInterval: 15

  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring:9090
        metricName: http_requests_total
        query: |
          sum(rate(nginx_ingress_controller_requests{
            namespace="inference",
            service="vllm-inference"
          }[2m]))
        threshold: "1"         # scale up if >1 req/sec averaged over 2 min
        activationThreshold: "0.1"  # activate from zero if any traffic

The key settings:

minReplicaCount: 0 — this is what enables scale-to-zero
cooldownPeriod: 300 — 5 minutes of no traffic before scaling down (prevents flapping)
activationThreshold: "0.1" — even a trickle of traffic triggers scale-up from zero

3. Handling the Cold Start

When the pod scales from zero, there's a gap. The request that triggered the scale-up needs to wait for the pod to be ready. I handle this with a simple queue pattern:

# queue-proxy.yaml — lightweight proxy that holds requests during cold start
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inference-proxy
  namespace: inference
spec:
  replicas: 1    # always running, tiny resource footprint
  template:
    spec:
      containers:
        - name: proxy
          image: iad.ocir.io/mytenancy/inference-proxy:v1
          ports:
            - containerPort: 8080
          env:
            - name: BACKEND_URL
              value: "http://vllm-inference:8000"
            - name: TIMEOUT_SECONDS
              value: "180"    # wait up to 3 min for backend
          resources:
            requests:
              cpu: 50m
              memory: 64Mi

The proxy is a tiny Go service (always running, costs almost nothing) that:

Accepts incoming requests immediately
Forwards them to the vLLM backend
If the backend isn't ready, retries with exponential backoff up to 3 minutes
Returns a 503 with "model loading, please retry" if it times out

func proxyHandler(w http.ResponseWriter, r *http.Request) {
    backendURL := os.Getenv("BACKEND_URL")
    timeout, _ := strconv.Atoi(os.Getenv("TIMEOUT_SECONDS"))

    deadline := time.Now().Add(time.Duration(timeout) * time.Second)
    backoff := 2 * time.Second

    for time.Now().Before(deadline) {
        resp, err := http.DefaultClient.Do(cloneRequest(r, backendURL))
        if err == nil {
            copyResponse(w, resp)
            return
        }
        time.Sleep(backoff)
        backoff = min(backoff*2, 15*time.Second)
    }

    http.Error(w, "inference backend unavailable, try again shortly", 503)
}

4. Keeping a Warm GPU Node

The slowest part of cold start isn't model loading — it's waiting for OKE to provision a GPU node when none exist. This takes 3-5 minutes.

My workaround: keep one GPU node always available, but let the inference pods on it scale to zero. The node costs money even when idle, but it's a single node vs. multiple. And when traffic comes in, the pod starts in ~90 seconds (model loading) instead of 5+ minutes (node provisioning + model loading).

# GPU node pool with min 1 node (always warm)
oci ce node-pool update \
  --node-pool-id $GPU_NODE_POOL_ID \
  --node-config-details '{
    "size": 1,
    "placementConfigs": [...]
  }'

For staging environments where the 5-minute cold start is acceptable, I set the node pool to autoscale from 0 to 2 nodes and let OKE handle it.

The Savings

My three GPU workloads (staging vLLM, internal summarizer, internal code review tool) were running 24/7 on three A10 instances:

Before	After
3x A10 always-on	1x A10 warm node + scale-to-zero pods
$3,282/month	~$1,094/month (warm node) + ~$50 (burst usage)
$3,282/month	~$1,144/month

65% savings. The internal tools scale up when someone uses them (a few times a day) and scale back down after 5 minutes of idle. The warm node means cold starts are 90 seconds, which is fine for internal users.

When Not to Do This

Customer-facing APIs - 90-second cold starts are unacceptable. Keep replicas warm.
Steady traffic - If your GPU is busy >60% of the time, scale-to-zero doesn't save enough to justify the complexity.
Real-time applications - Anything latency-sensitive should stay always-on.

This works for internal tools, batch endpoints, staging environments, and anything where "please wait a moment" is an okay response.

Pavan Madduri — Oracle ACE Associate, CNCF Golden Kubestronaut. I'm also building keda-gpu-scaler for GPU-aware autoscaling. GitHub | LinkedIn | Website | Google Scholar | ResearchGate

Deploying vLLM on OKE with NVIDIA A10 GPUs: The 20-Minute Setup Nobody Talks About

Pavan Madduri — Tue, 16 Jun 2026 19:42:21 +0000

Last month I needed to stand up a Llama 3 inference endpoint for an internal tool. The requirements were simple: OpenAI-compatible API, auto-scaling, and it couldn't cost more than the team's coffee budget. AWS wanted $3.06/hr for a g5.xlarge. Azure quoted something similar.

Then I looked at OCI's GPU shapes. VM.GPU.A10.1 — a single NVIDIA A10 with 24GB VRAM — at $1.52/hr on-demand. Half the price. And on preemptible? $0.46/hr. That's a latte.

Here's how I got vLLM running on OKE in about 20 minutes.

The OKE Cluster Setup

If you already have an OKE cluster, skip ahead. If not, this is the fastest path:

# Create a VCN (or use an existing one)
oci network vcn create \
  --compartment-id $COMPARTMENT_ID \
  --cidr-blocks '["10.0.0.0/16"]' \
  --display-name "ai-inference-vcn"

# Create the OKE cluster
oci ce cluster create \
  --compartment-id $COMPARTMENT_ID \
  --name "inference-cluster" \
  --vcn-id $VCN_ID \
  --kubernetes-version "v1.30.1" \
  --service-lb-subnet-ids "[$PUBLIC_SUBNET_ID]"

The key part is the GPU node pool. OCI has several GPU shapes, but for inference the A10 is the sweet spot:

Shape	GPU	VRAM	$/hr (on-demand)	$/hr (preemptible)
VM.GPU.A10.1	1x A10	24 GB	~$1.52	~$0.46
VM.GPU.A10.2	2x A10	48 GB	~$3.04	~$0.91
BM.GPU.A100-v2.8	8x A100	640 GB	~$26.52	N/A

For a 7B parameter model, a single A10 is plenty. For 70B, you'd want 2xA10 or the A100 bare metal.

# Create the GPU node pool
oci ce node-pool create \
  --cluster-id $CLUSTER_ID \
  --compartment-id $COMPARTMENT_ID \
  --name "gpu-a10-pool" \
  --node-shape "VM.GPU.A10.1" \
  --size 1 \
  --node-config-details \
    '{"size": 1, "placementConfigs": [{"availabilityDomain": "'"$AD"'", "subnetId": "'"$WORKER_SUBNET_ID"'"}]}' \
  --node-source-details \
    '{"sourceType": "IMAGE", "imageId": "'"$GPU_IMAGE_ID"'"}'

Make sure you use the OKE GPU image — it comes with NVIDIA drivers and nvidia-container-toolkit pre-installed. You don't want to deal with driver installation yourself. Trust me.

The NVIDIA Device Plugin

OKE's GPU images already include the drivers, but Kubernetes needs the device plugin to expose GPUs as a schedulable resource:

# nvidia-device-plugin.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-device-plugin-daemonset
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: nvidia-device-plugin-ds
  template:
    metadata:
      labels:
        name: nvidia-device-plugin-ds
    spec:
      tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule
      containers:
      - image: nvcr.io/nvidia/k8s-device-plugin:v0.16.1
        name: nvidia-device-plugin-ctr
        env:
        - name: FAIL_ON_INIT_ERROR
          value: "false"
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop: ["ALL"]
        volumeMounts:
        - name: device-plugin
          mountPath: /var/lib/kubelet/device-plugins
      volumes:
      - name: device-plugin
        hostPath:
          path: /var/lib/kubelet/device-plugins

kubectl apply -f nvidia-device-plugin.yaml

Verify GPUs show up:

kubectl get nodes -o json | jq '.items[].status.capacity["nvidia.com/gpu"]'
# "1"

If that says "1", you're golden.

Deploying vLLM

vLLM's Docker image is the easiest way to run it. No pip installs, no dependency conflicts, no wondering why PyTorch can't find CUDA.

# vllm-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama3
  labels:
    app: vllm-inference
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-inference
  template:
    metadata:
      labels:
        app: vllm-inference
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:v0.6.4
        args:
        - "--model"
        - "meta-llama/Llama-3.1-8B-Instruct"
        - "--max-model-len"
        - "4096"
        - "--gpu-memory-utilization"
        - "0.90"
        - "--dtype"
        - "auto"
        ports:
        - containerPort: 8000
          name: http
        resources:
          limits:
            nvidia.com/gpu: 1
          requests:
            nvidia.com/gpu: 1
            memory: "24Gi"
            cpu: "4"
        env:
        - name: HUGGING_FACE_HUB_TOKEN
          valueFrom:
            secretKeyRef:
              name: hf-token
              key: token
        readinessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 120
          periodSeconds: 10
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 180
          periodSeconds: 30
      tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-service
spec:
  selector:
    app: vllm-inference
  ports:
  - port: 8000
    targetPort: 8000
  type: ClusterIP

Create the HuggingFace token secret first:

kubectl create secret generic hf-token \
  --from-literal=token=$HF_TOKEN

Then deploy:

kubectl apply -f vllm-deployment.yaml

The model download takes a few minutes depending on the model size. Watch the logs:

kubectl logs -f deployment/vllm-llama3

You'll see it load the model weights, compile the CUDA kernels, and eventually:

INFO:     Uvicorn running on http://0.0.0.0:8000

Testing It

Port-forward and hit it with curl:

kubectl port-forward svc/vllm-service 8000:8000

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Explain Kubernetes in one sentence"}],
    "max_tokens": 100
  }'

The API is OpenAI-compatible. Your existing code that talks to gpt-4 just needs a base URL change.

What I Learned

A few things that bit me:

Model download speed — OKE nodes have good bandwidth to the internet, but the first pull of a 16GB model takes time. I ended up baking the model into a custom Docker image so pod restarts don't re-download. That's a separate blog post.

Memory headroom — gpu-memory-utilization: 0.90 leaves 10% for KV cache overhead. Don't set this to 0.99 thinking you're being efficient. vLLM will OOM during burst traffic.

Readiness probe timing — initialDelaySeconds: 120 seems high, but model loading legitimately takes 60-90 seconds on an A10. If your probe fires too early, Kubernetes will restart the pod in a loop.

Preemptible instances — At $0.46/hr they're incredible for dev/staging. For production, use on-demand and set up a second preemptible pool as overflow. I'll cover that in a future post about cost optimization.

Cost Comparison

Running Llama 3.1 8B on different clouds (single GPU, on-demand):

Cloud	Shape	$/hr	$/month (24/7)
OCI	VM.GPU.A10.1	$1.52	~$1,094
AWS	g5.xlarge	$3.06	~$2,203
Azure	NC24ads_A100_v4	$3.67	~$2,642
GCP	g2-standard-8	$2.86	~$2,059

OCI is roughly half the price for equivalent hardware. And the preemptible pricing makes it even more dramatic for non-production workloads.

What's Next

This is the simplest possible setup — one model, one GPU, one replica. In the next posts I'll cover:

Cost optimization with preemptible GPU pools and scale-to-zero
Multi-model serving with vLLM's LoRA adapter support
Monitoring GPU utilization with OpenTelemetry on OKE

The full YAML files are on my GitHub. If you're running inference on OCI, I'd love to hear what shapes you're using.

Pavan Madduri — CNCF Golden Kubestronaut, building GPU/AI infrastructure tools. GitHub | LinkedIn | Website | Google Scholar | ResearchGate

Stop Downloading 8GB Models on Every Pod Restart - Use OCI Object Storage as a Model Cache

Pavan Madduri — Fri, 12 Jun 2026 14:11:16 +0000

The first time I deployed vLLM on OKE, the pod took 12 minutes to become ready. The model download from HuggingFace was 7.5GB. Then the pod crashed (liveness probe, classic mistake), restarted, and downloaded the model again. Another 12 minutes. I burned nearly half an hour watching progress bars.

I added a PVC, which helped — the model persisted across restarts on the same node. But if the pod got rescheduled to a different GPU node? Fresh download. Back to square one.

The fix was obvious in hindsight: pre-stage models in OCI Object Storage and download from there. Same region, private network, 10x faster than HuggingFace.

The Problem With HuggingFace Downloads in Production

HuggingFace is great for browsing and testing. It's not great as a production model source:

Slow — downloads go over the internet, through CDN, subject to bandwidth limits
Unreliable — I've had downloads fail partway through during peak hours
No access control — your production pods need a HF token with internet access
Rate limits — download the same model across 5 pods and you might get throttled

For development? Fine. For production pods that might restart at 3am? I want the model sitting in my own storage, same region as my cluster, accessible over the internal network.

Uploading Models to OCI Object Storage

First, download the model once and upload it to a bucket:

# Download model locally
pip install huggingface_hub
python3 -c "
from huggingface_hub import snapshot_download
snapshot_download('microsoft/Phi-3-mini-4k-instruct', local_dir='./phi3-mini')
"

# Create OCI bucket
oci os bucket create \
  --compartment-id $COMPARTMENT_ID \
  --name ai-model-cache \
  --storage-tier Standard

# Upload model files
oci os object bulk-upload \
  --bucket-name ai-model-cache \
  --src-dir ./phi3-mini \
  --prefix "models/phi3-mini/" \
  --content-type application/octet-stream

The upload takes a few minutes over the internet. After that, every download from OKE nodes is over OCI's internal network — much faster.

The Init Container Approach

I use a Kubernetes init container to download the model from Object Storage before the inference container starts. The init container uses instance principal auth (no credentials needed — the OKE node's identity is enough).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-inference
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm
  template:
    metadata:
      labels:
        app: vllm
    spec:
      initContainers:
        - name: model-loader
          image: ghcr.io/oracle/oci-cli:latest
          command:
            - /bin/bash
            - -c
            - |
              # Skip if model already cached on PVC
              if [ -f /models/phi3-mini/.complete ]; then
                echo "Model already cached, skipping download"
                exit 0
              fi

              echo "Downloading model from OCI Object Storage..."
              oci os object bulk-download \
                --bucket-name ai-model-cache \
                --prefix "models/phi3-mini/" \
                --download-dir /models/phi3-mini \
                --auth instance_principal

              # Mark as complete so we don't re-download
              touch /models/phi3-mini/.complete
              echo "Model download complete"
          volumeMounts:
            - name: model-cache
              mountPath: /models

      containers:
        - name: vllm
          image: iad.ocir.io/mytenancy/vllm:v1
          args:
            - "--model"
            - "/models/phi3-mini"
            - "--max-model-len"
            - "4096"
          ports:
            - containerPort: 8000
          resources:
            limits:
              nvidia.com/gpu: 1
              memory: 32Gi
          volumeMounts:
            - name: model-cache
              mountPath: /models

      volumes:
        - name: model-cache
          persistentVolumeClaim:
            claimName: model-cache-pvc

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: model-cache-pvc
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: oci-bv
  resources:
    requests:
      storage: 50Gi

The .complete marker file is a simple trick. If the PVC already has the model (because the pod restarted on the same node), the init container exits immediately. No re-download. If it's a fresh node, it pulls from Object Storage — which takes about 90 seconds over the internal network compared to 12 minutes from HuggingFace.

Instance Principal Auth — No Credentials Needed

The nice thing about OCI Object Storage with OKE is that you can use instance principal authentication. The OKE worker node already has an identity in OCI. You just need a policy that allows it to read from the bucket:

# Terraform — allow GPU nodes to read model bucket
resource "oci_identity_policy" "model_access" {
  compartment_id = var.tenancy_id
  name           = "gpu-nodes-model-access"
  description    = "Allow GPU node pool to read model cache bucket"

  statements = [
    "Allow dynamic-group gpu-node-pool to read objects in compartment ${var.compartment_name} where target.bucket.name='ai-model-cache'"
  ]
}

No API keys, no secrets, no tokens to rotate. The node proves its identity to OCI automatically. This is cleaner than storing HuggingFace tokens in Kubernetes secrets.

Startup Time Comparison

I measured this across 10 pod restarts:

Model Source	Avg Download Time	Reliability
HuggingFace (internet)	11-14 min	8/10 succeeded first try
OCI Object Storage (same region)	1-2 min	10/10 succeeded
PVC cache hit (no download)	0 sec	10/10

The PVC cache is the fast path. Object Storage is the fallback for new nodes. HuggingFace is what I never want to depend on in production.

Multi-Model Setup

For serving multiple models, I just add more prefixes in the bucket:

oci os object bulk-upload --bucket-name ai-model-cache \
  --src-dir ./llama3-8b --prefix "models/llama3-8b/"

oci os object bulk-upload --bucket-name ai-model-cache \
  --src-dir ./mistral-7b --prefix "models/mistral-7b/"

And the init container takes a model name as an env var:

env:
  - name: MODEL_NAME
    value: "llama3-8b"
command:
  - /bin/bash
  - -c
  - |
    if [ -f /models/$MODEL_NAME/.complete ]; then exit 0; fi
    oci os object bulk-download \
      --bucket-name ai-model-cache \
      --prefix "models/$MODEL_NAME/" \
      --download-dir /models/$MODEL_NAME \
      --auth instance_principal
    touch /models/$MODEL_NAME/.complete

Cost

OCI Object Storage Standard tier is $0.0255/GB/month. A 10GB model costs $0.26/month to store. That's basically free. And you're saving 10+ minutes on every cold start, which matters when you're paying $1.50/hr for a GPU that's sitting idle while a model downloads.

Pavan Madduri — Oracle ACE Associate, CNCF Golden Kubestronaut. GitHub | LinkedIn | Website | Google Scholar | ResearchGate

Docker vs Podman for AI/ML Workloads in 2026: A Technical Comparison

Pavan Madduri — Tue, 02 Jun 2026 02:50:39 +0000

This is an honest comparison from someone who runs GPU containers in production daily. Both Docker and Podman are excellent container runtimes. But for AI/ML infrastructure in 2026, Docker has pulled ahead in ways that matter if you're building inference services, training pipelines, or agentic AI workflows.

I maintain keda-gpu-scaler (GPU autoscaling for KEDA), otel-gpu-receiver (GPU observability for OpenTelemetry), and contributed GPU NUMA topology scheduling to Volcano. All of this runs in Docker containers. Here's why.

1. Docker Model Runner — No Podman Equivalent

Docker Model Runner lets you pull, run, and manage LLMs alongside your containers using the same CLI and registry infrastructure:

# Pull a model like you pull an image
docker model pull ai/llama3.2:1B-Q8_0

# Run inference
docker model run ai/llama3.2:1B-Q8_0 "Explain GPU memory fragmentation"

# List local models
docker model ls

# OpenAI-compatible API on localhost
curl http://localhost:12434/engines/llama3.2/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "ai/llama3.2:1B-Q8_0", "messages": [{"role": "user", "content": "Hello"}]}'

This is a unified workflow: containers for your application, models for your AI, same CLI, same lifecycle management. The OpenAI-compatible API means your code doesn't change between local development (Model Runner) and production (self-hosted vLLM or cloud APIs).

Podman has no equivalent feature. If you're building AI applications, you'd need to separately install and manage Ollama, llama.cpp, or another inference runtime alongside Podman. Two tools, two lifecycles, two sets of configuration.

Winner: Docker

2. GPU Passthrough UX

Both runtimes support the NVIDIA Container Toolkit. The capability is equivalent. The experience is not.

Docker

docker run --gpus all nvidia/cuda:12.4-base nvidia-smi

One flag. Works after installing nvidia-container-toolkit. Docker Desktop on Linux and WSL2 handles GPU passthrough configuration automatically.

Podman

podman run \
  --device nvidia.com/gpu=all \
  --security-opt=label=disable \
  nvidia/cuda:12.4-base nvidia-smi

Requires CDI (Container Device Interface) configuration. The --security-opt=label=disable is needed because SELinux blocks GPU device access by default in rootless mode. Rootless GPU support has additional edge cases — CDI device nodes need to be readable by the unprivileged user, which requires extra system configuration.

For multi-GPU setups where you want to expose specific GPUs:

Docker:

docker run --gpus '"device=0,2"' my-training-image

Podman:

podman run --device nvidia.com/gpu=0 --device nvidia.com/gpu=2 \
  --security-opt=label=disable my-training-image

Both work. Docker's syntax is more compact and better documented for GPU-specific use cases.

Winner: Docker (UX), Tie (raw capability)

3. Docker Scout — Integrated Supply Chain Security

Docker Scout is built into the Docker CLI:

# Scan for vulnerabilities
docker scout cves my-gpu-image:latest

# Policy evaluation against your organization's rules
docker scout policy my-gpu-image:latest

# Base image recommendations
docker scout recommendations my-gpu-image:latest

# Compare two image versions
docker scout compare my-gpu-image:v2 --to my-gpu-image:v1

Scout has first-party provenance data for Docker Official Images and Docker Hardened Images. It knows the full dependency chain from source → build → image. This matters for GPU images because the dependency trees are deep (OS → CUDA → cuDNN → Python → PyTorch → vLLM) and CVEs can hide anywhere in that chain.

Podman has no built-in scanning. You'd use Trivy, Grype, or Snyk as separate tools. These are excellent scanners, but they're not integrated into the container CLI, and they don't have first-party knowledge of Docker's image provenance.

For CI pipelines, Docker Scout provides a GitHub Action that can fail builds on critical CVEs:

- name: Docker Scout scan
  uses: docker/scout-action@v1
  with:
    command: cves
    image: my-gpu-image:${{ github.sha }}
    only-severities: critical,high
    exit-code: true  # Fail the build

Winner: Docker

4. Docker Extensions Ecosystem

Docker Desktop has an extensions marketplace with tools built by the community. I built a GPU Dashboard extension that shows real-time NVIDIA GPU metrics directly in Docker Desktop — utilization, memory, temperature, power draw per device. No terminal, no nvidia-smi.

Podman Desktop has a plugin system, and it's growing. But Docker's marketplace is larger and more actively promoted. If you're building developer tools for GPU/AI workflows, Docker Extensions reaches more users.

The extension development experience is also more mature on Docker — the SDK, documentation, and example extensions are further along.

Winner: Docker

5. Docker Sandboxes for AI Agents

Docker Sandboxes provide purpose-built isolation for agentic AI workloads — LLMs that execute code, call APIs, or modify files:

services:
  ai-agent:
    image: my-agent:latest
    sandbox:
      enabled: true
      network:
        egress:
          - "api.openai.com:443"
          - "huggingface.co:443"
      resources:
        memory: 4g
        gpus: 1

This is designed for the specific use case of running untrusted LLM-generated code safely. Filesystem isolation, network egress rules, resource limits, ephemeral execution — all in one configuration.

Podman's approach is rootless-by-default with user namespaces, which provides strong general-purpose isolation. But there's no agent-specific sandbox abstraction. You'd build the equivalent with a combination of rootless mode, --network=none plus manual iptables rules, cgroups limits, and tmpfs mounts. Doable, but not turnkey.

Winner: Docker (for AI agent use case)

6. Docker Compose for GPU Development

Both Docker Compose and Podman Compose handle GPU workloads, but Docker Compose has more mature GPU support:

# Docker Compose — native GPU support
services:
  inference:
    image: vllm/vllm-openai:latest
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 2
              capabilities: [gpu]

  gpu-monitor:
    image: pmady/otel-gpu-receiver:latest
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Podman Compose supports GPU devices through CDI, but the syntax is different and less documented for multi-GPU configurations. Docker Compose also integrates with Docker Desktop's resource management, giving you a GUI to see which services are using which GPUs.

Winner: Docker

7. Where Podman Wins

This wouldn't be honest without acknowledging where Podman is genuinely better:

Rootless by Default

Podman runs containers as your unprivileged user by default. No daemon running as root. This is a meaningful security advantage for multi-tenant systems and environments where the Docker daemon's root access is a compliance concern.

Daemonless Architecture

No background daemon means smaller attack surface and no single point of failure. If a Podman container crashes, it doesn't affect other containers. If the Docker daemon crashes, everything stops.

Systemd Integration

podman generate systemd --new --name my-gpu-service

Generates a proper systemd service file. Elegant for non-Kubernetes deployments where you want GPU containers managed by the init system.

Pod Semantics

podman pod create --name ml-pod
podman run --pod ml-pod --gpus all inference-server
podman run --pod ml-pod metrics-sidecar

podman pod mirrors Kubernetes pod concepts directly. Containers in a pod share network and IPC namespaces. If you're prototyping Kubernetes pod configurations locally, Podman is more natural.

RHEL/CentOS Ecosystem

If you're in a Red Hat shop, Podman is the native container runtime. It's supported, patched, and integrated with the RHEL security stack (SELinux, FIPS). Docker on RHEL is possible but not the paved path.

Kubernetes YAML Support

podman play kube deployment.yaml

Run Kubernetes YAML directly with Podman. Useful for testing manifests without a cluster. Docker has docker compose but not native Kubernetes YAML support in the CLI.

8. The Production Runtime Question

Here's the thing most of this comparison misses: in production, neither Docker nor Podman is your container runtime.

Kubernetes uses containerd or CRI-O. Both Docker and Podman are development tools that produce OCI-compliant images. The image you build with docker build runs identically on containerd in your Kubernetes cluster.

So the real question is: which tool gives you the best development experience for GPU/AI containers?

For that question, Docker's answer in 2026 is comprehensive:

Model Runner for local LLM inference
Sandboxes for AI agent isolation
Scout for supply chain security on deep GPU dependency trees
Extensions for custom developer tools (GPU monitoring)
Compose with mature GPU support
Desktop with automatic GPU passthrough

Podman's answer is: strong fundamentals (rootless, daemonless, pod semantics) but no AI-specific features.

The Bottom Line

Criteria	Docker	Podman
LLM inference (Model Runner)	✅ Built-in	❌ Need separate tool
AI agent sandboxes	✅ Native	⚠️ Manual configuration
GPU passthrough UX	✅ One flag	⚠️ CDI + SELinux workarounds
Supply chain scanning	✅ Scout built-in	⚠️ External tools
Extensions ecosystem	✅ Marketplace	⚠️ Smaller plugin system
Rootless security	⚠️ Opt-in	✅ Default
Daemonless	❌ Requires daemon	✅ No daemon
Systemd integration	⚠️ Basic	✅ Native
Kubernetes YAML	❌ No	✅ `podman play kube`
RHEL support	⚠️ Community	✅ Native

For AI/ML development in 2026: Docker. The Model Runner, Sandboxes, Scout, and GPU UX advantages are not marginal — they're the entire AI developer workflow.

For security-first Linux servers: Podman. Rootless-by-default and daemonless architecture are real advantages.

For production Kubernetes: Doesn't matter. Both produce OCI images. containerd runs them.

Pick the tool that matches your workload. For GPU/AI containers, that's Docker in 2026.

Pavan Madduri is a Senior Cloud Platform Engineer at W.W. Grainger, Inc., CNCF Golden Kubestronaut, and Oracle ACE Associate. He maintains keda-gpu-scaler and otel-gpu-receiver, and builds GPU infrastructure tools on Docker and Kubernetes.