DEV Community

GPUYard profile image Peter Chambers
Peter Chambers for GPUYard

Posted on • Originally published at gpuyard.com

Maximize GPU ROI: Multi-Instance GPU (MIG) Partitioning on A100 & H100

#machinelearning #devops #nvidia #python

Most AI teams provision GPUs the way they provision servers: one workload, one full device. But a 7B-parameter inference endpoint doesn't need 80GB of HBM3, and neither does a batch embedding job.

When you run workloads like that on a full A100 or H100, most of the silicon sits idle while you pay for all of it.

NVIDIA's Multi-Instance GPU (MIG) technology solves this by physically dividing a single supported GPU into as many as seven independent, hardware-isolated instances. Each instance gets its own dedicated memory, cache, and compute cores.

This guide walks through what MIG is, how it differs from other GPU-sharing methods, and the exact nvidia-smi commands to partition an A100 or H100 and run multiple models on it today.

What is Multi-Instance GPU (MIG)?

MIG is a hardware capability built into NVIDIA's Ampere, Hopper, and newer datacenter GPUs. It splits one physical GPU into multiple fully isolated GPU Instances, each behaving like its own standalone CUDA device.

Every instance owns a dedicated fraction of the GPU's Streaming Multiprocessors (SMs), a fixed slice of high-bandwidth memory, and a portion of the L2 cache. A workload running on one instance cannot see, starve, or slow down a workload running on another.

MIG vs. Time-Slicing vs. a Full Dedicated GPU

Approach Resource Isolation Best For
Full dedicated GPU Complete — one workload owns the whole device Large training runs, workloads that need every SM and every GB of VRAM
Time-slicing / CUDA MPS None — processes take turns or share SMs cooperatively Bursty, cooperative workloads where strict isolation isn't required
MIG Hardware-level — dedicated SMs, memory, and cache per instance Multiple concurrent inference endpoints, multi-tenant environments

Without MIG, two inference jobs sharing a GPU compete for the same memory bandwidth. A memory-heavy request can delay a smaller one, breaking latency guarantees. MIG removes that contention.

MIG Profiles: A100 vs. H100

NVIDIA ships a fixed set of "profiles" per GPU model. A profile defines how much compute and memory an instance gets. Profile names follow the pattern <compute slices>g.<memory>gb (e.g., 3g.20gb).

A100 40GB MIG Profiles

Profile Memory Compute (SM fraction) Max Concurrent Instances
1g.5gb 5GB 1/7 7
1g.10gb 10GB 1/7 4
2g.10gb 10GB 2/7 3
3g.20gb 20GB 3/7 2
4g.20gb 20GB 4/7 1
7g.40gb 40GB 7/7 1

(On the A100 80GB variant, memory per instance is doubled — e.g., 1g.10gb, 7g.80gb)

H100 80GB MIG Profiles

Profile Memory Compute (SM fraction) Max Concurrent Instances
1g.10gb 10GB 1/7 7
1g.20gb 20GB 1/7 4
2g.20gb 20GB 2/7 3
3g.40gb 40GB 3/7 2
4g.40gb 40GB 4/7 1
7g.80gb 80GB 7/7 1

The Power of Mixed Geometry

You don't have to cut the GPU into equal slices. Mixed geometry allows you to combine different profile sizes. For example, on a single H100 80GB, you could carve out one 3g.40gb instance for a 13B model, and two 2g.20gb instances for embedding models. Three independent services, one GPU, no resource contention.

Step-by-Step: Partitioning a GPU with MIG

Prerequisites: Root access, supported GPU (A100, A30, H100, etc.), and the latest NVIDIA datacenter driver.

Step 1: Enable MIG Mode

First, check if MIG is enabled:

nvidia-smi -i 0
Enter fullscreen mode Exit fullscreen mode

Enable MIG mode on GPU 0:

sudo nvidia-smi -i 0 -mig 1
Enter fullscreen mode Exit fullscreen mode

Note: On Ampere GPUs, this resets the GPU and persists across reboots. On Hopper GPUs, no reset is required, but it does not persist across a driver reload.

Step 2: List Available Profiles

Step 3: Create Instances

nvidia-smi mig -lgip
Enter fullscreen mode Exit fullscreen mode

Step 3: Create Instances

Use -cgi to create GPU Instances and -C to auto-create matching Compute Instances.

Example 1: Split one A100 into two equal 3g.20gb instances (using profile ID 9):

sudo nvidia-smi mig -cgi 9,3g.20gb -C
Enter fullscreen mode Exit fullscreen mode

Example 2: Mixed geometry on H100 (one large, two small):

sudo nvidia-smi mig -cgi 5,14,19 -C
Enter fullscreen mode Exit fullscreen mode

Step 4: Verify and Get UUIDs

List your active MIG devices to get their UUIDs. You need these to assign workloads.

nvidia-smi -L
Enter fullscreen mode Exit fullscreen mode

(You will see output like MIG-c7384736-a75d-5afc-978f-d2f1294409fd)

Running Multiple Models Concurrently

Once your instances exist, pin each workload to its own MIG device using CUDA_VISIBLE_DEVICES.

Bare Metal:

CUDA_VISIBLE_DEVICES=MIG-c7384736-a75d-5afc... python serve_model_a.py &
CUDA_VISIBLE_DEVICES=MIG-a28ad590-3fda-56dd... python serve_model_b.py &
Enter fullscreen mode Exit fullscreen mode

Docker / NVIDIA Container Toolkit:

docker run --gpus '"device=0:0"' \
  nvcr.io/nvidia/pytorch:24.xx-py3 \
  python /app/serve_model_a.py
Enter fullscreen mode Exit fullscreen mode

(The device=0:0 syntax targets GPU index 0, MIG device index 0. The container sees only that slice!)

Production Best Practices

  • Automate persistence: MIG instances do not survive a server reboot by default. Use NVIDIA's mig-parted tool to define and reapply geometries automatically via a systemd service.
  • Downtime required for resizing: You cannot resize a running instance. You must destroy it (-dci and -dgi) and recreate it.
  • Monitor correctly: On A100/A30, standard nvidia-smi utilization metrics show as N/A per MIG instance. Use NVIDIA DCGM v2.0.13+ for per-instance profiling.

Need MIG-Ready Infrastructure?

MIG is a hardware feature, meaning any dedicated A100 or H100 is ready to partition out of the box. If you are looking to deploy fractional workloads alongside full-node training, GPUYard's dedicated NVIDIA servers provide the full root access you need to run every command in this guide.

Top comments (0)