DEV Community: Manikandan T

The GPU Cold Starts Nobody Warns You About: Autoscaling LLM Inference on Kubernetes

Manikandan T — Wed, 03 Jun 2026 18:23:20 +0000

The Problem

When an LLM inference pod scales on Kubernetes, it doesn't start in seconds like a CPU service. It hits a sequential chain of bottlenecks that can stall Time-to-First-Token (TTFT) for 10+ minutes:

Each phase blocks the next. Optimizing one shifts the bottleneck downstream - you must address all four.

The reason these traps hurt so much is that a GPU inference pod doesn't come up in seconds like a stateless web service. It moves through a sequential chain where each stage blocks the next:

Modern inference containers (vLLM + PyTorch + CUDA + NCCL) are 10–20GB. Model weights for production models (Llama-3 70B, DeepSeek-V3) exceed 130GB. This is the reality of GPU autoscaling - every cold start moves tens of gigabytes before generating a single token.

This post covers practical solutions for each phase, issues I hit during implementation, and cloud-native alternatives across providers.

Phase 1: Node Provisioning

Karpenter Over Cluster Autoscaler

If you're still on Cluster Autoscaler for inference workloads, switch to Karpenter. The key differences that matter for GPU scaling:

Event-driven - reacts to pending pods in milliseconds vs CA's 10s+ polling loop
Direct cloud API - bypasses ASGs/VMSS entirely, selects instance types dynamically
Workload-aware - evaluates GPU, memory, CPU, taints, affinity constraints together and picks the optimal instance

For inference specifically, the important Karpenter configuration is restricting your NodePool to a consistent GPU family. This matters downstream for compile cache sharing (covered in Phase 4).

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu-inference
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]
        - key: node.kubernetes.io/instance-type
          operator: In
          values: ["p4d.24xlarge"]  # A100 instances - consistent GPU architecture
      taints:
        - key: nvidia.com/gpu
          value: "true"
          effect: NoSchedule
  disruption:
    consolidationPolicy: WhenEmpty
    consolidateAfter: 5m

Scaling Strategy: Scale When N Pods Are Running

Instead of warm pools with pause pods, configure your HPA/KEDA to trigger node provisioning proactively when existing pods approach saturation. The idea: when you have N pods running and load reaches a threshold, scale to N+1 before all pods are saturated.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-scaler
spec:
  scaleTargetRef:
    name: vllm-inference
  minReplicaCount: 1
  maxReplicaCount: 8
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus:9090
        metricName: vllm_pending_requests
        query: sum(vllm:num_requests_waiting)
        threshold: "10"  # Scale when queue depth exceeds threshold

This ensures Karpenter sees a pending pod early, begins provisioning, and the new pod is ready before existing pods are overwhelmed. You never scale from zero - you scale from N to N+1.

The hard floor remains: VM allocation + OS boot + GPU driver initialization + kubelet registration = 90–120 seconds for GPU instances. GPU driver init is the hidden cost - NVIDIA driver + device enumeration adds 20–30s that CPU instances don't pay. The only way past this floor is having the node already running.

Cloud-Specific Provisioning

Phase 2: Container Image Pull

The Problem

A vLLM container image is 10–20GB. When Karpenter provisions a fresh node, downloading from DockerHub/ECR/ACR saturates NAT gateways. Multiple nodes pulling simultaneously = ImagePullBackOff errors and 3-5 minute delays.

Why Lazy Pulling (eStargz/SOCI) Fails for GPU Workloads

Lazy pulling restructures images to allow on-demand file extraction via FUSE. The container "starts" immediately and fetches files as needed via HTTP Range Requests.

Why this is catastrophic for ML inference: Python ML runtimes import thousands of shared objects sequentially (torch, libcudart, triton, etc.). Each uncached import = synchronous HTTP round-trip through FUSE. The result: pull time drops dramatically, but application readiness (time to first successful /health response) regresses badly - often by an order of magnitude or more. The latency shifts from a one-time bulk download to thousands of small, serialized network round-trips spread across the Python import sequence.

The registry becomes a runtime dependency. Every import torch call blocks on network I/O. Do not use lazy pulling for LLM inference.

Spegel: P2P Image Distribution

Spegel is the practical P2P solution for GPU clusters - stateless, lightweight, zero control plane overhead.

How it works:

DaemonSet on every node (must tolerate GPU taints)
Advertises SHA256 layer digests via Kademlia DHT
containerd configured to route pulls through localhost mirror
New nodes query DHT, stream layers from peers over internal VPC network
404 fallback to external registry if layer not found in cluster

helm upgrade --install spegel oci://ghcr.io/spegel-org/helm-charts/spegel \
  --namespace spegel --create-namespace \
  --version v0.7.1 \
  --set "spegel.mirrorResolveTimeout=5s" \
  --set "spegel.mirrorResolveRetries=5" \
  --set "tolerations[0].key=nvidia.com/gpu" \
  --set "tolerations[0].operator=Exists" \
  --set "tolerations[0].effect=NoSchedule"

Critical requirement: For Spegel to work, at least one node must already have the image cached (this is why you scale from N to N+1, not from zero).

Compatibility:
Refer - https://spegel.dev/docs/getting-started/#compatibility

Issue: Default mirrorResolveTimeout Is Too Aggressive
Spegel's default mirrorResolveTimeout is 20ms. This is extremely tight Kademlia DHT lookups that exceed 20ms fall back to the upstream registry, even when a peer has the layer. This explains why you might see ~90% hit rates instead of near-100%. Increasing to 5s with 5 retries gives the P2P network enough time to resolve peers.

Issue: containerd 2.1 Breaks Spegel Silently
This was the most time-consuming debugging issue. If you're running AL2023, Ubuntu 24.04, or any OS with containerd 2.1, there are three breaking defaults:

1. use_local_image_pull = false (default)
containerd 2.1 routes all pulls through a new transfer service (io.containerd.transfer.v1). This transfer service does NOT honor registry mirrors in hosts.toml. Spegel is silently bypassed - every pull goes to the external registry regardless of mirror configuration.

2. discard_unpacked_layers = true (default)
containerd discards compressed layers after extraction. Spegel needs preserved layers to serve them to peers. Without them, the P2P network fails silently.

3. Registry-specific hosts.toml overrides _default/hosts.toml
If you create docker.io/hosts.toml in your node userData, it overrides Spegel's _default/hosts.toml mirror configuration. Spegel's init container creates the _default/hosts.toml - do NOT create registry-specific host files.

The fix - containerd overrides in node userData:

# Example: EC2NodeClass userData for AL2023
userData: |
  MIME-Version: 1.0
  Content-Type: multipart/mixed; boundary="BOUNDARY"

  --BOUNDARY
  Content-Type: application/node.eks.aws

  ---
  apiVersion: node.eks.aws/v1alpha1
  kind: NodeConfig
  spec:
    containerd:
      config: |
        [plugins."io.containerd.cri.v1.images".registry]
          config_path = "/etc/containerd/certs.d"
        [plugins."io.containerd.cri.v1.images"]
          discard_unpacked_layers = false
          use_local_image_pull = true
  --BOUNDARY--

Cloud-Native Image Solutions

These outperform P2P for most use cases:

Verdict: Cloud-native streaming > Spegel P2P > Lazy pulling. Use Spegel for multi-cloud or when cloud-native options are unavailable.

Phase 3: Model Weight Loading

The Math

T_transfer = Payload_Size / Bandwidth
130 GB over 10 Gbps = ~104 seconds (theoretical minimum)

Real-world transfers from S3/HuggingFace Hub: 3–5 minutes due to TCP overhead and rate limiting.

Shared Storage (PVC) for Pre-Downloaded Weights
The standard pattern: pre-download weights to a ReadWriteMany PVC, mount it in inference pods.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: model-cache-pvc
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: efs-sc  # Or Azure Files / Filestore
  resources:
    requests:
      storage: 200Gi

Pre-download via Job:

apiVersion: batch/v1
kind: Job
metadata:
  name: model-download
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: downloader
          image: python:3.11-slim
          command: ["sh", "-c"]
          args:
            - |
              pip install huggingface_hub[hf_xet]
              huggingface-cli download meta-llama/Llama-3-70B-Instruct \
                --local-dir /shared/models/Llama-3-70B-Instruct
          env:
            - name: HF_TOKEN
              valueFrom:
                secretKeyRef:
                  name: hf-token
                  key: token
          volumeMounts:
            - name: shared-cache
              mountPath: /shared
      volumes:
        - name: shared-cache
          persistentVolumeClaim:
            claimName: model-cache-pvc

Limitation: Shared filesystems (EFS/NFS/Azure Files) have IOPS bottlenecks when many nodes read 130GB concurrently. EFS elastic mode delivers ~130 MB/s - that's 15x slower than local NVMe.

Node-Local NVMe: The Performance Path

A100 instances (p4d/p5 on AWS, ND-series on Azure, A2/A3 on GKE) include physically attached NVMe disks delivering ~2,000+ MB/s reads. Use them.

NVMe Setup: Approaches by Environment
Option 1: Karpenter instanceStorePolicy (recommended for EKS)
Karpenter's EC2NodeClass supports instanceStorePolicy: RAID0, which auto-formats all NVMe instance store devices as RAID0 and mounts them to /mnt/k8s-disks/0. Kubelet uses this as ephemeral storage, so emptyDir volumes are automatically NVMe-backed. No userData script, no privileged pods, and no manual device detection.

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: gpu-nodeclass
spec:
  amiSelectorTerms:
    - alias: al2023@latest
  instanceStorePolicy: RAID0
  # ... other fields (role, subnets, security groups, etc.)

This is the cleanest approach for Karpenter one line in the EC2NodeClass spec. The NVMe is ready before any pod schedules, and using emptyDir instead of hostPath means no hardcoded mount paths and no Pod Security Policy concerns.

Option 2: userData/cloud-init
For environments without Karpenter (self-managed ASGs, other provisioners), format and mount NVMe during node boot via userData:

# EC2NodeClass userData - formats NVMe at boot
userData: |
  MIME-Version: 1.0
  Content-Type: multipart/mixed; boundary="BOUNDARY"

  --BOUNDARY
  Content-Type: text/x-shellscript

  #!/bin/bash
  # Format and mount instance store NVMe (skip boot volume nvme0)
  DEVICES=$(ls /dev/nvme*n1 2>/dev/null | grep -v nvme0)
  if [ -n "$DEVICES" ]; then
    DEVICE_COUNT=$(echo "$DEVICES" | wc -l)
    if [ "$DEVICE_COUNT" -gt 1 ]; then
      yum install -y mdadm || apt-get install -y mdadm
      echo "y" | mdadm --create /dev/md0 --level=0 \
        --raid-devices=$DEVICE_COUNT $DEVICES
      mkfs.ext4 -F /dev/md0
      mkdir -p /mnt/fast-disks
      mount /dev/md0 /mnt/fast-disks
    else
      mkfs.ext4 -F $DEVICES
      mkdir -p /mnt/fast-disks
      mount $DEVICES /mnt/fast-disks
    fi
    chmod 777 /mnt/fast-disks
  fi

  --BOUNDARY
  Content-Type: application/node.eks.aws
  # ... rest of NodeConfig (containerd overrides etc.)
  --BOUNDARY--

Note: AL2023 uses dnf, not yum. This approach still avoids privileged DaemonSets and Pod Security Policy violations.

Option 3: Cloud-managed NVMe provisioning
Azure Container Storage auto-provisions NVMe RAID0 StoragePools on ND-series VMs (similar to Karpenter's instanceStorePolicy). GKE local SSDs can be configured via node pool settings.

Option 4: Privileged DaemonSet (development/testing only)
A DaemonSet with privileged: true and hostPID: true can format NVMe drives post-boot. However, this is typically blocked in production by Pod Security Standards (Restricted/Baseline), OPA/Gatekeeper/Kyverno policies, and compliance requirements (SOC2, PCI-DSS). Only use this in development clusters where security policies are relaxed.

InitContainer: PVC → NVMe Sync
Once NVMe is available (via any method above), use an initContainer to copy the pre-downloaded model from the shared PVC to NVMe-backed storage:

initContainers:
  - name: model-sync
    image: alpine:latest
    command: ["sh", "-c"]
    args:
      - |
        echo "Syncing model from shared PVC to NVMe-backed storage..."
        cp -r /shared/models/Llama-3-70B-Instruct /nvme/models/
        echo "Sync complete."
    volumeMounts:
      - name: shared-cache
        mountPath: /shared
        readOnly: true
      - name: nvme-storage
        mountPath: /nvme

With instanceStorePolicy: RAID0, mount the NVMe-backed volume as emptyDir: {} (kubelet places it on the NVMe mount automatically). With the userData approach, use hostPath: { path: /mnt/fast-disks }. The emptyDir approach is preferred because it avoids hardcoded paths, works with Pod Security Standards, and Kubernetes manages the lifecycle.

Each pod pays the PVC to NVMe copy cost (~60–90s for 130GB at EFS elastic throughput). With emptyDir, each pod copies independently (emptyDir is per-pod), but the copy from EFS to NVMe is still far faster than reading directly from EFS during inference.

Cloud-Specific Model Storage

GKE's Hyperdisk ML is notable: a single pre-populated volume mounted read-only across 2,500 pods eliminates all multi-node download redundancy.

NVIDIA ModelExpress / NIXL (Experimental)

Note: I have not tested this personally. The following is from NVIDIA documentation and community reports. Including it because it represents the theoretical fastest path for weight distribution in multi-node GPU clusters.

For environments with RDMA/InfiniBand interconnects (multi-node A100/H100 clusters), NVIDIA ModelExpress enables P2P weight distribution:

New worker communicates with metadata server (Redis sidecar)
Locates active GPU worker running the same model
Streams tensors directly from active worker's GPU memory via RDMA
Zero storage dependency for scale-out

# UCX transport configuration for NIXL
UCX_TLS=rc_x,rc,dc_x,dc,cuda_copy
UCX_RNDV_SCHEME=get_zcopy
MODEL_EXPRESS_NO_SHARED_STORAGE=1  # gRPC fallback when shared storage unavailable

This is the fastest possible weight distribution - active GPU memory to new GPU memory over InfiniBand. Relevant for large TP deployments on H100 clusters with EFA/InfiniBand interconnects.

Phase 4: fastsafetensors and Weight Loading Optimization

Standard Loading Path

The CPU RAM acts as a bounce buffer - data passes through without computation, purely as a transfer intermediary. For 130GB of weights on A100 with 80GB VRAM, this means multiple sequential PCIe transfers with CPU orchestration. Multi-minute load times.

fastsafetensors: Faster Weight Loading

fastsafetensors provides two loading paths:

1. GDS path: On GDS-optimized distributed filesystems (Lustre, WekaFS), it uses NVIDIA GPUDirect Storage to DMA directly from storage to GPU VRAM, bypassing CPU entirely. Performance: 4.8x to 7.5x speedup over standard loading.

2. POSIX I/O path (nogds): On local NVMe/ext4 or when GDS drivers are unavailable, it falls back to an optimized POSIX I/O path. This is still significantly faster than standard loading when reading from NVMe (~2,000+ MB/s) vs shared filesystems like EFS (~130 MB/s).
The key insight: the biggest performance gain comes from NVMe vs shared filesystem, not from GDS bypassing the CPU. Moving model weights from EFS to local NVMe is a ~15x bandwidth improvement regardless of whether GDS is active.

Enable in vLLM:

args:
  - "--load-format"
  - "fastsafetensors"
env:
  - name: USE_FASTSAFETENSOR
    value: "true"

GDS: When It Matters and When It Doesn't
GDS (GPUDirect Storage) enables direct DMA from storage to GPU VRAM, bypassing CPU bounce buffers. But GDS is a filesystem-level optimization, not just a driver. It requires a GDS-optimized filesystem to function:

GDS-optimized filesystems: Lustre (FSx for Lustre), WekaFS, GPFS these support the cuFile API natively. On these, fastsafetensors delivers 4.8–7.5x speedup via true DMA.

Local NVMe/ext4: Not GDS-optimized. Even with nvidia_fs.ko loaded, GDS runs in compatibility mode (CPU bounce buffer, no faster than POSIX I/O) or doesn't engage at all. fastsafetensors detects this and falls back to its nogds path.

How to check GDS status on your node:

# Check if nvidia-gds package is installed
dpkg -l | grep nvidia-gds    # Debian/Ubuntu
rpm -qa | grep nvidia-gds     # RHEL/AL2023

# Check if GDS kernel module is loaded
lsmod | grep nvidia_fs
# From inside a container, check vLLM's detection
# Look for this in logs:
# "GDS not enabled, setting nogds=True"  ← GDS NOT available
# No such message = GDS is active

GDS Compatibility by Instance:
GDS support depends on the driver stack (nvidia-fs kernel module, MOFED/OFED drivers) and critically, the storage filesystem. Any modern NVIDIA datacenter GPU can technically do GDS if the correct drivers are installed AND the filesystem supports the cuFile API.

Issue I hit: On g6 instances (L4, testing a smaller model), vLLM logged GDS not enabled, setting nogds=True. The AL2023 GPU AMI does not include nvidia-gds. But even installing GDS would not have helped here the model was on local NVMe/ext4, which is not a GDS-optimized filesystem. The real fix was moving model weights from EFS to NVMe, not installing GDS drivers.

On datacenter instances (p4d/p5/ND-series/A3) the MOFED + nvidia-fs stack is pre-installed. fastsafetensors delivers its full 4.8–7.5x GDS speedup only when reading from a GDS-optimized distributed filesystem (e.g., FSx for Lustre). When reading from local NVMe/ext4 on these same instances, the GDS bypass does not engage the speedup comes from NVMe bandwidth, not GDS.

NUMA Binding for Multi-Socket A100/H100 Servers
On p4d (8x A100) and p5 (8x H100), the machine has multiple NUMA domains. Without binding, Python workers may drift across CPU sockets, causing cross-NUMA memory access during Tensor Parallel sharding.
The standard approach is to wrap vLLM with numactl or configure NUMA affinity via environment variables:

env:
  - name: VLLM_WORKER_MULTIPROC_METHOD
    value: "spawn"  # Required for multi-GPU - default 'fork' causes issues with CUDA contexts

For explicit NUMA pinning, wrap the entrypoint with numactl:

command: ["numactl", "--cpunodebind=0", "--membind=0", "python3", "-m", "vllm.entrypoints.openai.api_server"]

This pins each GPU worker to the CPU cores and memory closest to its PCIe lanes. Primarily affects steady-state throughput rather than startup latency, but prevents throughput degradation under TP configurations.

Note: Some vLLM versions expose a --enable-numa flag. Verify availability in your target version - the NUMA interface has changed across releases. The VLLM_WORKER_MULTIPROC_METHOD=spawn env var is the stable requirement for multi-GPU setups.

Phase 5: torch.compile Cache Persistence

The Problem
After weights are loaded, vLLM uses torch.compile to trace CUDA graphs for the decode loop. TorchInductor generates low-level device kernels optimized for the specific GPU architecture.

Default FULL_AND_PIECEWISE mode captures:

Monolithic decode graphs for uniform sequence lengths
Segmented piecewise graphs for variable prefill dimensions

This compilation is strictly serial, CPU-bound. Every new pod pays this penalty independently. Compilation time scales with model size and sequence length range: a 7B model on a single GPU compiles in ~20–30s, while a 70B model across 8x A100 with large context ranges can take 2–5 minutes.

The Fix: Shared Compile Cache on RWX Storage
Redirect the compile cache to shared storage. First pod compiles, all subsequent pods reuse.

env:
  - name: VLLM_CACHE_ROOT
    value: "/shared/compile_cache"  # Points to RWX PVC (EFS/Azure Files/Filestore)

Cache structure:
/shared/compile_cache/
  torch_compile_cache/
    <hash>/                         # model config + PyTorch version + GPU compute capability
      rank_0_0/
        backbone/
          transformed_code.py       # Compiled Python code
          computation_graph.py      # Graph structure
          inductor_cache/           # Final compiled kernels

The cached artifacts are environment-specific - they're safe to reuse only across pods sharing the same GPU architecture, CUDA, PyTorch, and vLLM version. vLLM derives this by hashing a long list of config and PyTorch factors, and the underlying Inductor kernels are compiled to architecture-specific cubins - so the binding to a specific GPU is real even though it isn't a single tidy "compute capability" field. First pod writes the cache; subsequent pods in the same environment hit and load directly. This is vendor-confirmed: vLLM's docs state the compiled artifacts can be reused across machines with the same environment, and explicitly recommend generating the cache once and sharing it among instances for autoscaling.

Result from implementation (7B model, single L4 GPU, max_model_len=12000):

Log evidence of a successful hit:

That's a 60% reduction on a 7B model from a single environment variable. For larger models (70B on 8x A100), fresh compilation can take 2–5 minutes - the cache hit savings scale proportionally, typically reducing to 10–20s of cache loading.

Issue: GPU Architecture Specificity

The compile cache hash includes GPU compute capability:

A100 (compute 8.0) → cache hash: 8d22bdd77e
H100 (compute 9.0) → cache hash: f04cb94f7b

An A100 cache CANNOT be reused on an H100 node. If your NodePool allows mixed GPU types, pods on different architectures face full recompilation despite cache existing.
Solution: Restrict NodePool to a single GPU family:

requirements:
  - key: node.kubernetes.io/instance-type
    operator: In
    values: ["p4d.24xlarge"]  # All nodes = A100, same compute capability

This is why Phase 1 matters - consistent GPU architecture enables compile cache sharing cluster-wide.

Compilation Mode Reference

Complete Optimized Deployment
Putting it all together - vLLM on A100 with all optimizations:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-optimized
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-optimized
  template:
    metadata:
      labels:
        app: vllm-optimized
    spec:
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      initContainers:
        - name: model-sync
          image: alpine:latest
          command: ["sh", "-c"]
          args:
            - |
              echo "Copying model from shared PVC to NVMe-backed emptyDir..."
              cp -r /shared/models/Llama-3-70B-Instruct /nvme/models/
          volumeMounts:
            - name: shared-cache
              mountPath: /shared
              readOnly: true
            - name: nvme-storage
              mountPath: /nvme
      containers:
        - name: vllm
          image: vllm/vllm-openai:latest
          command: ["numactl", "--cpunodebind=0", "--membind=0",
                    "python3", "-m", "vllm.entrypoints.openai.api_server"]
          args:
            - "--model"
            - "/nvme/models/Llama-3-70B-Instruct"
            - "--tensor-parallel-size"
            - "8"
            - "--load-format"
            - "fastsafetensors"
            - "--gpu-memory-utilization"
            - "0.90"
            - "--dtype"
            - "bfloat16"
            - "--host"
            - "0.0.0.0"
            - "--port"
            - "8000"
          env:
            - name: USE_FASTSAFETENSOR
              value: "true"
            - name: VLLM_WORKER_MULTIPROC_METHOD
              value: "spawn"
            - name: VLLM_CACHE_ROOT
              value: "/shared/compile_cache"
          resources:
            requests:
              nvidia.com/gpu: "8"
              cpu: "96"
              memory: "1024Gi"
            limits:
              nvidia.com/gpu: "8"
          volumeMounts:
            - name: nvme-storage
              mountPath: /nvme
              readOnly: true
            - name: shared-cache
              mountPath: /shared
          startupProbe:
            httpGet:
              path: /health
              port: 8000
            failureThreshold: 120
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            failureThreshold: 3
            periodSeconds: 5
      volumes:
        - name: nvme-storage
          emptyDir: {}  # NVMe-backed via instanceStorePolicy: RAID0
        - name: shared-cache
          persistentVolumeClaim:
            claimName: model-cache-pvc

Note: The nvme-storage volume uses emptyDir: {} which is automatically NVMe-backed when the EC2NodeClass has instanceStorePolicy: RAID0. For non-Karpenter environments using userData-formatted NVMe, replace with hostPath: { path: /mnt/fast-disks }.

Cloud-Native Reference Architectures

AWS EKS

Karpenter (EC2NodeClass) → p4d.24xlarge (8x A100)
  → Bottlerocket + EBS Snapshot (zero-second image pull)
  → EFS PVC (compile cache, RWX)
  → NVMe hostPath (model weights, PCIe speed)
  → vLLM + fastsafetensors + VLLM_CACHE_ROOT

Azure AKS

NAP (AKSNodeClass) → ND A100 v4
  → ACR Artifact Streaming (~5s image availability)
  → Azure Container Storage (NVMe RAID0, auto-provisioned)
  → Azure Files PVC (compile cache, RWX)
  → vLLM + fastsafetensors + VLLM_CACHE_ROOT

Google GKE

ComputeClasses (NAP) → A2/A3 (A100/H100)
  → GKE Image Streaming (transparent remote mount)
  → Hyperdisk ML (model weights, ReadOnlyMany, 2500 pods)
  → GCS/Filestore PVC (compile cache, RWX)
  → vLLM + fastsafetensors + VLLM_CACHE_ROOT

Comparison

Expected Timings (Theoretical)

With all optimizations on A100 infrastructure (p4d.24xlarge), scaling from N to N+1 pods where N >= 1:

Notes on these numbers:

Node provisioning 90–120s includes: EC2/VM API call (~5s) + instance allocation (~15–30s) + OS boot (~15s) + NVIDIA driver initialization (~20–30s) + kubelet registration + node ready (~10–15s). GPU driver init is the hidden cost most people underestimate - it adds 20–30s that CPU instances don't pay.
Spegel P2P 30–60s: A 15–20GB image over VPC internal networking (p4d has 100 Gbps baseline). Spegel uses HTTP-based transfer over TCP, not RDMA - real throughput is well below wire speed. DHT lookups and multi-peer coordination add overhead. Pin to v0.7.1 and tune mirrorResolveTimeout to 5s for reliable hit rates
Cloud-native streaming 5–15s: EBS Snapshot = data pre-attached at boot (near-instant). ACR/GKE streaming = remote filesystem mount with on-demand paging (container starts before full download).
PVC → NVMe sync 60–90s: 130GB at EFS elastic throughput (~1.5 GB/s burst). Shared filesystem read speed is the bottleneck here. On Azure with premium NVMe StoragePool, or with S3/Blob direct download, this can be faster.
Weight loading 15–25s: 130GB / 8 GPUs = ~16GB per GPU. PCIe Gen4 x16 = 32 GB/s per GPU theoretical. With safetensors metadata parsing overhead, effective throughput is ~40–60% of theoretical. On local NVMe/ext4, fastsafetensors uses POSIX I/O (not GDS) GDS only engages on distributed filesystems like Lustre.
Compile cache load 10–20s for 70B: Loading pre-compiled graphs from shared storage for all TP ranks. Larger models have more compilation units to deserialize.

Lessons Learned

Node Provisioning

Restrict NodePool to a single GPU instance type - this enables compile cache sharing across nodes
Karpenter's security groups on provisioned nodes may differ from managed node groups - cross-SG ingress rules are needed for pod networking
Set consolidateAfter appropriately - too aggressive and you lose nodes that could serve the next scale event

Image Pull (Spegel / P2P)

containerd 2.1 on AL2023/Ubuntu 24.04 defaults use_local_image_pull=false - Spegel is silently bypassed
discard_unpacked_layers=true breaks P2P layer serving - must be explicitly overridden
Registry-specific docker.io/hosts.toml overrides Spegel's _default/hosts.toml - do not create registry-specific host files in userData
Spegel requires at least one existing node with the cached image - useless for true scale-from-zero, design around scale-from-N
Default mirrorResolveTimeout of 20ms is too aggressive Kademlia DHT lookups exceeding this fall back to upstream. Tune to 5s with 5 retries for better hit rates
Large base layers (PyTorch/CUDA) may not have distribution source labels in the image manifest, causing DHT lookup failures - these fall back to registry pulls

Model Weight Loading

EFS/NFS throughput (~130 MB/s) is slower than local NVMe (~2000 MB/s) - use NVMe for weight reads, shared PVC for pre-download and cache sharing. Throughput may vary based on file system used.
The initContainer PVC → NVMe copy is a one-time cost per node - subsequent pods on the same node find the model on NVMe and skip it
For 130GB+ models, PVC → NVMe copy takes 60–90s - this is unavoidable on first boot but is a one-time cost
Use Karpenter's instanceStorePolicy: RAID0 for NVMe provisioning (preferred) or userData/cloud-init as a fallback. Avoid privileged DaemonSets.

fastsafetensors / GDS

The primary speedup is NVMe vs shared filesystem (~15x bandwidth) not GDS bypassing the CPU
nogds=True in vLLM logs is expected and not a problem on local NVMe/ext4 fastsafetensors' POSIX I/O path is still fast
GDS only provides its 4.8–7.5x speedup on GDS-optimized distributed filesystems (Lustre, WekaFS, GPFS) not on local ext4/XFS
Even with nvidia-gds installed and nvidia_fs loaded, GDS runs in compatibility mode on ext4 (CPU bounce buffer, same as POSIX)
Datacenter instances (p4d/p5/ND-series/A3) ship with MOFED + nvidia-fs pre-installed but GDS only engages when paired with a GDS-optimized filesystem
For local NVMe setups, fastsafetensors is still worth using its optimized POSIX I/O path is efficient on NVMe

torch.compile Cache

Highest ROI optimization - one env var (VLLM_CACHE_ROOT), trivial to implement, 60%+ compile time reduction
Cache is GPU-architecture-specific - A100 cache (8.0) != H100 cache (9.0), never mix in same NodePool
First pod after a PyTorch version upgrade or model config change will recompile - cache invalidation is hash-based
The shared PVC must be ReadWriteMany - first pod writes, subsequent pods read

General

Solve every phase - optimizing one just exposes the next bottleneck
Cloud-native solutions (image streaming, Hyperdisk ML) give the biggest wins with least operational complexity
The physical limit is PCIe bandwidth + VM boot time - everything else is software-solvable
Scale from N >= 1, not from zero - this enables Spegel P2P, warm NVMe caches, and avoids the worst-case cold start

Repository
The infrastructure code, Kubernetes manifests, and deployment configurations referenced in this post are available in the companion repository:
GitHub Repository - GPU Autoscaling Accelerator

This post synthesizes research from multiple design iterations and hands-on implementation on EKS with Karpenter, Spegel, fastsafetensors, and persistent torch.compile caching.

72B Parameters, Zero Quantization, One GPU: Benchmarking Qwen2-VL on AMD MI300X

Manikandan T — Wed, 13 May 2026 08:02:13 +0000

I loaded Qwen2-VL-72B-Instruct at full BF16 precision on a single GPU, served 64 concurrent DocVQA streams, and kept the system stable at 99.5% KV cache utilization - all for $1.99/hour on the AMD Developer Cloud.

This post walks through exactly how I did it: the hardware economics that make it possible, the deployment configuration that makes it stable, and the benchmark results that prove it works.

Why This Matters

Building enterprise-grade visual RAG architectures - Invoice extraction, contract intelligence, automated RFP processing, document QA, OCR-heavy PDF understanding, and long-context retrieval pipelines - requires vision-language models that don't hallucinate structural details. Qwen2-VL-72B is still one of the most capable open-weights models for these tasks.

The problem is running it. A 72-billion parameter model in BF16 precision consumes roughly 144GB of VRAM just to load the weights. Traditional 80GB GPUs force you into aggressive 4-bit quantization, which severely degrades OCR accuracy and multimodal reasoning.

The AMD Instinct MI300X changes the deployment calculus entirely. With 192GB of HBM3 memory, it fits the full unquantized model on a single GPU and leaves 48GB of headroom for KV caches and concurrent workloads.

The Economics: MI300X vs. A100 and H100

Before diving into deployment details, let's address the cost question — because hardware costs cannot be evaluated in a vacuum. You have to evaluate the cost per usable gigabyte of VRAM required to serve your specific model.

The NVIDIA 80GB Constraint

If you deploy on NVIDIA infrastructure using A100 (80GB) or H100 (80GB) GPUs, a single GPU is physically incapable of loading Qwen2-VL-72B unquantized. You are forced into one of two compromises.

The first option is aggressive quantization: crush the model down to 4-bit (AWQ/GPTQ) to fit it on a single 80GB card. This severely degrades OCR and multimodal reasoning capabilities — exactly the capabilities you need for enterprise document processing.

The second option is tensor parallelism (TP=2): provision a multi-GPU node and shard the model across two cards using --tensor-parallel-size 2. This works, but it introduces cross-device NCCL communication overhead on every forward pass, inflating inter-token latency beyond what the raw memory bandwidth would suggest.

The Cost Breakdown

Using standard tier-2 cloud pricing (Lambda Cloud, CoreWeave - generally cheaper than AWS/GCP on-demand):

A 2x A100 (80GB) node runs approximately $3.00 to $4.00 per hour per card. You get the 160GB of pooled VRAM you need, but on older Ampere architecture with slower memory bandwidth, plus the NCCL overhead between cards.

A 2x H100 (80GB) node runs approximately $6.00 to $8.00+ per hour per card. Hopper is blazing fast, but you are paying for two cards' worth of compute just to get 160GB of pooled VRAM - and you still carry the TP=2 communication overhead.

A single AMD MI300X (192GB) node on the AMD Developer Cloud costs $1.99 per hour (Price may vary for production).

The Architectural Advantage

The MI300X doesn't just cut the hourly cost by 50-75%. It completely eliminates the complexity of multi-GPU tensor parallelism. There is no cross-device communication overhead. The inter-token latency is bounded strictly by the 5.3 TB/s memory bandwidth of a single HBM3 pool — and my stress test benchmarks confirmed ITL of 43-66ms at the synchronous baseline, which validates that the memory subsystem delivers on its theoretical bandwidth promises.

For enterprise teams scaling visual RAG pipelines, this shifts the unit economics of multimodal inference from prohibitive to profitable.

Hardware and Environment

I provisioned the environment on the AMD Developer Cloud. Here are the system specifications:

GPU: 1x AMD Instinct MI300X (192GB HBM3 VRAM)

Compute: 20 vCPUs, 240GB RAM

Boot Storage: 720GB NVMe SSD

Scratch Storage: 5TB NVMe SSD

Software Stack: Ubuntu 22.04, ROCm 7.2.0, Docker

The 192GB VRAM is the critical specification. With ~144GB consumed by the model weights, that leaves approximately 48GB of headroom. That 48GB is what allows processing massive base64-encoded images, maintaining large context windows, and handling concurrent batch requests without triggering OOM errors.

Preparing NVMe Storage

The 5TB NVMe scratch disk needs to be mounted for the HuggingFace cache. Downloading 144GB of weights to the boot disk will exhaust space and throttle loading times.

# Format the scratch disk with XFS (excellent large-file and parallel I/O handling)
sudo wipefs -af /dev/vdc1
sudo mkfs.xfs -f /dev/vdc1
sudo mkdir -p /mnt/models
sudo mount /dev/vdc1 /mnt/models
sudo chown -R $USER:$USER /mnt/models

# Point HuggingFace cache to the NVMe drive
mkdir -p /mnt/models/huggingface
export HF_HOME=/mnt/models/huggingface
echo "export HF_HOME=/mnt/models/huggingface" >> ~/.bashrc

With the cache on NVMe, subsequent container restarts load the full 144GB of weights into VRAM in seconds rather than minutes.

Deploying vLLM on MI300X

Deploying vLLM on AMD hardware requires passing the correct kernel drivers into the Docker container. Unlike NVIDIA's --gpus all flag, the ROCm ecosystem requires direct device passthrough of the KFD (Kernel Fusion Driver) and DRI (Direct Rendering Infrastructure) interfaces.

Here is the production deployment command:

docker run -d \
  --name vllm-qwen2-vl-72b \
  --network host \
  --ipc=host \
  --device=/dev/kfd \
  --device=/dev/dri \
  --group-add video \
  --group-add render \
  -v /mnt/models:/mnt/models:rw \
  -e HF_HOME=/mnt/models/huggingface \
  -e VLLM_USE_TRITON_FLASH_ATTN=1 \
  --model Qwen/Qwen2-VL-72B-Instruct \
  vllm/vllm-openai-rocm:v0.20.1 \
  --dtype bfloat16 \
  --gpu-memory-utilization 0.92 \
  --max-model-len 16384 \
  --max-num-seqs 64 \
  --max-num-batched-tokens 8192 \
  --enable-chunked-prefill \
  --port 8000

Why Each Flag Matters

Host integration (--network host --ipc=host): Bypassing Docker's bridge network eliminates overhead, which is critical for benchmarking true API latency. Host IPC is required for efficient shared memory operations between vLLM's internal processes.

ROCm passthrough (--device=/dev/kfd --device=/dev/dri --group-add video --group-add render): This is how the container communicates with the CDNA architecture of the MI300X. If your container fails to start with mysterious ROCm errors, the cause is almost always a permissions issue with these device paths or missing group additions.

Precision (--dtype bfloat16): BF16 is the optimal datatype for MI300X. It provides the same dynamic range as FP32, preventing the numerical overflow issues that occur with standard FP16 during the massive attention matrix multiplications in 72B+ models. The MI300X Matrix Core technology natively supports BF16 — do not force FP16 on this architecture.

Memory management (--gpu-memory-utilization 0.92): This tells vLLM to reserve 92% of the 192GB VRAM. After loading the model weights, the remaining allocation is dedicated entirely to the KV cache block pool, managed by vLLM's PagedAttention system. The engine carved out 32.18 GiB specifically for the KV cache, providing 105,440 tokens of cache capacity.

Concurrency limits (--max-num-seqs 64, --max-num-batched-tokens 8192): These define the batching boundaries to prevent OOM under heavy load. With 64 maximum concurrent sequences and 8192 tokens per batch, the scheduler has enough room to interleave requests without exhausting the KV cache blocks.

Chunked prefill (--enable-chunked-prefill): This is non-negotiable for multimodal models. Vision inputs generate massive prompt token counts — a single high-resolution document image can tokenize into thousands of visual tokens. Without chunked prefill, a single massive document would monopolize the entire prefill pipeline, stalling all other requests in the batch. Chunked prefill breaks the initial prompt processing into smaller chunks and interleaves them with decode steps from other in-flight requests.

Because vLLM exposes an OpenAI-compatible API, this endpoint is a drop-in replacement for existing application logic. LangChain, LlamaIndex, or custom agentic workflows can point directly to localhost:8000/v1 without modifying the integration layer.

Monitoring AMD GPUs During Inference

If you come from the NVIDIA ecosystem, your muscle memory will reach for nvidia-smi. On consumer AMD cards, you might try radeontop. Neither works for data center CDNA architectures like the MI300X.

The correct tool is amd-smi (or rocm-smi):

watch -n 2 amd-smi

Two things to note from this output. First, the VRAM usage stays relatively static during inference because vLLM pre-allocates the entire KV cache block pool at startup based on the 0.92 utilization flag. What fluctuates is power draw and GPU utilization, which spike during the compute-heavy prefill phases of multimodal requests. Second, amd-smi sometimes aggregates memory differently than nvidia-smi. Trust the vLLM engine logs — specifically the "GPU KV cache usage" percentage reported every 10 seconds — for the most accurate view of your KV cache block utilization.

Benchmarking the Deployment

To validate this infrastructure for production document processing, I used GuideLLM to run two distinct benchmarking phases against the live endpoint.

Phase A: Synthetic Stress Test — VRAM Saturation Sweep

This test was designed to push the KV cache to its absolute breaking point using maximum-context synthetic prompts.

guidellm benchmark \
  --target "http://localhost:8000/v1" \
  --model "Qwen/Qwen2-VL-72B-Instruct" \
  --profile sweep \
  --data "prompt_tokens=8192,output_tokens=1024" \
  --max-seconds 300 \
  --warmup 10 \
  --output-dir ./results-stress \
  --outputs json,html

The sweep profile automatically escalates from synchronous (1 request at a time) through throughput-maximizing batches and then across increasing constant-rate loads. This produces a full performance curve from idle to saturated.

The critical result from Phase A: the synchronous baseline ITL was 39.6ms (median), climbing to 66.8ms at higher concurrency. This proves the MI300X HBM3 memory bandwidth is delivering. Anything under 100ms ITL feels instantaneous to a human reader in a streaming interface.

At peak load, the KV cache hit 99.5% utilization — and the system survived. This is where chunked prefill earns its keep. Without it, sending a massive batch of new prompts to a system at 99% KV cache capacity would cause an immediate OOM crash. Chunked prefill allows the scheduler to break incoming prefill work into small blocks, filling the remaining gaps without exceeding physical limits.

Phase B: Enterprise DocVQA Workload

Synthetic data validates the infrastructure. Real data validates the architecture. I used the lmms-lab/DocVQA dataset, throwing 64 concurrent streams at the GPU to simulate a heavily loaded internal document analysis tool.

guidellm benchmark \
  --target "http://localhost:8000/v1" \
  --model "Qwen/Qwen2-VL-72B-Instruct" \
  --profile concurrent \
  --rate 64 \
  --data "lmms-lab/DocVQA" \
  --data-args '{"name": "DocVQA"}' \
  --max-seconds 120 \
  --warmup 10 \
  --output-dir ./results-doc \
  --outputs json,html

The DocVQA results tell a different story than the synthetic test — and that's the point. Real multimodal workloads are fundamentally harder than synthetic text. Each document image tokenizes into thousands of visual tokens (median 4,996 input tokens per request, with 3.86 million pixels of image data), which means the prefill phase dominates. The median TTFT of 38.5 seconds at 64 concurrent streams reflects the GPU working through massive vision encoder computations for dozens of simultaneous documents.

The system completed 46 requests in 110 seconds with 64 concurrent streams — no errors, no OOMs, no crashes. The server throughput of 2,621 total tokens per second demonstrates that even under extreme multimodal concurrency, the architecture remains stable.

Understanding the Latency Pipeline

To build reliable systems on top of these numbers, you need to understand what happens between the moment a user submits a request and the moment they see the complete response. The inference pipeline has two fundamentally different computational phases, and each one is bottlenecked by a different hardware resource.

TTFT: Time To First Token (Compute-Bound)

TTFT measures the time between request submission and the first generated token appearing. For multimodal models, TTFT is dominated by the prefill phase — the GPU must process the base64 image through the vision encoder (ViT), project the visual embeddings into the LLM's token space, concatenate them with the text prompt tokens, and then perform the full self-attention computation over the entire combined sequence to populate the KV cache.

This is a compute-bound operation. The GPU cores are doing dense matrix multiplications across thousands of visual tokens. Under the 64-stream DocVQA load, TTFT was 38.5 seconds (median) — each request is competing for compute time with 63 other in-flight prefill and decode operations.

In production, if TTFT is too high for your SLA, the levers are: reduce --max-num-seqs to limit concurrency (trading throughput for latency), tune --max-num-batched-tokens to prioritize individual request latency, or scale horizontally by adding more MI300X nodes behind a load balancer.

ITL: Inter-Token Latency (Memory-Bandwidth-Bound)

Once prefill completes and the KV cache is populated, the model enters the decode phase. It generates one token at a time in an autoregressive loop. Each token generation requires reading the entire 144GB of model weights from HBM3 VRAM to the compute units.

This is a memory-bandwidth-bound operation. The GPU cores are fast enough — they are waiting on data delivery from memory. This is why the MI300X's 5.3 TB/s HBM3 bandwidth matters so much. My synthetic stress test showed a synchronous ITL baseline of 39.6ms, which aligns closely with the theoretical minimum: 144GB of weights divided by 5.3 TB/s bandwidth equals roughly 27ms per token, with the remainder accounted for by attention computation over the KV cache, kernel launch overhead, and scheduling latency.

At higher concurrency, ITL rises because the memory bus is shared across all in-flight decode operations. Under the synthetic sweep, ITL scaled gracefully from 39.6ms (synchronous) to 66.8ms (highest constant rate) — a 1.7x increase despite a 6x increase in concurrency. Under the DocVQA workload at 64 concurrent streams, the median ITL was 1,879ms, reflecting the extreme memory pressure of simultaneously maintaining KV caches for 64 high-resolution document contexts.

How Chunked Prefill Prevents Catastrophic Failure

During Phase A, the KV cache hit 99.5% utilization. Without chunked prefill, a new incoming request at this point would attempt to allocate its full prefill budget in one shot — and fail with an OOM crash, potentially taking down the entire serving process.

Chunked prefill changes this behavior. Instead of processing the entire prompt in a single monolithic computation, the scheduler breaks the prefill into smaller chunks (bounded by --max-num-batched-tokens). Between chunks, it interleaves decode steps from other in-flight requests. This means the system can gradually allocate KV cache blocks as they become available from completed requests, rather than demanding the full allocation upfront. The result is graceful degradation under pressure rather than catastrophic failure.

Practical Lessons Learned

VRAM reporting nuances. The amd-smi tool and the VRAM bar visualization sometimes report different figures than what vLLM's internal engine logs show. This is because amd-smi reports total GPU memory allocation (including driver overhead, CUDA graphs, and pre-allocated buffers), while vLLM reports specifically on KV cache block utilization. For production monitoring, instrument against the vLLM /metrics Prometheus endpoint, which exposes vllm:gpu_cache_usage_perc directly.

The BF16 imperative. Do not attempt FP16 on MI300X for models of this size. BF16 is natively supported by the Matrix Core technology, maintains FP32-equivalent dynamic range, and avoids the precision loss that causes output degradation in 72B+ parameter models. This is not a preference — it is a correctness requirement.

ROCm is production-ready. The ROCm 7.2 + vLLM v0.20.1 stack ran stable through sustained stress testing with zero crashes. For teams evaluating AMD as an alternative to NVIDIA for inference workloads, the ecosystem has matured significantly. The primary friction point is in the initial Docker configuration (device passthrough and group permissions), not in runtime stability.

SHM sizing. If you encounter cross-process communication errors in vLLM, pass --shm-size 8g to your Docker run command. This is not always required but resolves intermittent failures in certain multi-worker configurations.

Reproduce This

The exact commands used in this post:

# 1. Mount NVMe and set HF cache
sudo wipefs -af /dev/vdc1 && sudo mkfs.xfs -f /dev/vdc1
sudo mkdir -p /mnt/models && sudo mount /dev/vdc1 /mnt/models
sudo chown -R $USER:$USER /mnt/models
mkdir -p /mnt/models/huggingface
export HF_HOME=/mnt/models/huggingface

# 2. Launch vLLM (ROCm, v0.20.1)
docker run -d --name vllm-qwen2-vl-72b \
  --network host --ipc=host \
  --device=/dev/kfd --device=/dev/dri \
  --group-add video --group-add render \
  -v /mnt/models:/mnt/models:rw \
  -e HF_HOME=/mnt/models/huggingface \
  -e VLLM_USE_TRITON_FLASH_ATTN=1 \
  vllm/vllm-openai-rocm:v0.20.1 \
  --model Qwen/Qwen2-VL-72B-Instruct \
  --dtype bfloat16 \
  --gpu-memory-utilization 0.92 \
  --max-model-len 16384 \
  --max-num-seqs 64 \
  --max-num-batched-tokens 8192 \
  --enable-chunked-prefill \
  --port 8000

# 3. Stress test (synthetic sweep)
guidellm benchmark \
  --target "http://localhost:8000/v1" \
  --model "Qwen/Qwen2-VL-72B-Instruct" \
  --profile sweep \
  --data "prompt_tokens=8192,output_tokens=1024" \
  --max-seconds 300 \
  --warmup 10 \
  --output-dir ./results-stress \
  --outputs json,html

# 4. DocVQA benchmark (64 concurrent streams)
guidellm benchmark \
  --target "http://localhost:8000/v1" \
  --model "Qwen/Qwen2-VL-72B-Instruct" \
  --profile concurrent \
  --rate 64 \
  --data "lmms-lab/DocVQA" \
  --data-args '{"name": "DocVQA"}' \
  --max-seconds 120 \
  --warmup 10 \
  --output-dir ./results-doc \
  --outputs json,html

Final Thoughts

The AMD Instinct MI300X fundamentally alters how we architect enterprise AI infrastructure. Loading a 72-billion parameter multimodal model with zero quantization, dedicating 32GB to the KV cache, and serving 64 concurrent document analysis streams on a single node at $1.99/hour - this is a capability that did not exist at this price point 12 months ago.

For ML engineering teams building automated document processing, visual data extraction, or complex agentic systems, the VRAM constraints of 80GB hardware have forced painful compromises between model quality and deployment feasibility. The MI300X, paired with ROCm 7.2 and vLLM's advanced scheduling (chunked prefill, PagedAttention), provides a stable, powerful foundation for production-grade unquantized inference - at a fraction of the cost of equivalent NVIDIA configurations.

And AMD is continuing to push the memory boundary further. The Instinct MI325X extends capacity to 256GB HBM3E, targeting massive MoE and ultra-long-context inference workloads. Beyond that, the Instinct MI350X and MI355X move into next-generation CDNA4 territory with 288GB HBM3E, positioning AMD aggressively for frontier-scale enterprise AI.

What makes this trajectory especially significant is not just raw capacity - it is architectural simplification. Today, deploying a 72B model unquantized on NVIDIA means splitting weights across multiple GPUs, engineering around KV cache exhaustion, and accepting the latency overhead of cross-device communication. With 192GB-class accelerators, those constraints disappear for this model class. With 256–288GB, they disappear for even larger architectures - MoE models, ultra-long-context workloads, and multi-modal pipelines that would currently require four or more 80GB cards.

For enterprise AI engineering, the shift from 80GB-class to 192–288GB-class accelerators is not incremental. It fundamentally changes what becomes practical in production: fewer nodes, simpler serving topologies, lower operational complexity, and - critically - no quantization tax on model quality.