Mustafa ERBAY

Posted on Aug 8 • Originally published at mustafaerbay.com.tr

GPU Sharing in Kubernetes: The MIG vs. Time-Slicing Decision

#kubernetes #ai

Efficiently sharing GPU resources among multiple workloads in Kubernetes clusters is a critical step to optimize cost-effectiveness and performance. Two main approaches stand out for this sharing: NVIDIA's Multi-Instance GPU (MIG) technology and software-based Time-Slicing methods. Both methods offer different levels of isolation and flexibility, necessitating careful consideration of workload requirements when making a choice.

In this article, I will pragmatically discuss the fundamental operating principles, advantages, disadvantages, and suitable scenarios for both MIG and Time-Slicing. My goal is to provide a concrete decision guide for those looking to optimize GPU utilization in Kubernetes-based AI/ML infrastructures.

Why is GPU Resource Sharing Necessary?

GPUs have become indispensable hardware, especially for deep learning, machine learning, and high-performance computing (HPC) workloads. However, modern GPUs require significant investment and often cannot be fully utilized by a single small workload. For example, a lightweight AI inference model might only use a small fraction of an expensive A100 GPU.

This situation leads to underutilization of GPUs and resource waste. Orchestration platforms like Kubernetes are designed to efficiently manage server resources, but we need specific sharing mechanisms for GPUs. The fact that different workloads (training, inference, batch processing) have different GPU requirements increases the importance of a shared infrastructure.

Effective GPU sharing, on one hand, maximizes returns on hardware investments, and on the other hand, offers developers faster access and a flexible working environment. Especially in multi-tenant environments, having a shared pool rather than each user or team having their own GPU reduces operational complexity and lowers costs.

What is NVIDIA MIG (Multi-Instance GPU) and How Does It Work?

NVIDIA MIG (Multi-Instance GPU) is a technology that allows certain NVIDIA GPUs to be partitioned into multiple independent GPU instances at the hardware level. MIG is supported on GPUs starting with the NVIDIA Ampere architecture (A100, A30), Hopper architecture (H100, H200), and Blackwell architecture (B200, GB200, RTX PRO 6000 Blackwell series). This ensures that each MIG instance behaves like a completely isolated GPU with its own cores, L2 cache, memory bandwidth, and dedicated memory.

MIG offers hardware-guaranteed isolation for each "GPU instance." This means that a MIG instance assigned to a pod will not have any negative impact on the performance or security of other MIG instances. Performance fluctuations and "noisy neighbor" effects are largely eliminated with MIG. This feature is particularly important for multi-tenant and critical workloads.

To use MIG in a Kubernetes environment, the NVIDIA Device Plugin must be configured correctly. The Device Plugin detects MIG-enabled GPUs on the node and exposes each MIG instance as a separate Kubernetes resource. Pods can then request these specific resources in the format nvidia.com/mig-<size>.<memory> (e.g., nvidia.com/mig-1g.5gb).

apiVersion: v1
kind: Pod
metadata:
  name: mig-pod
spec:
  restartPolicy: Never
  containers:
    - name: cuda-container
      image: nvcr.io/nvidia/cuda:11.4.0-runtime-ubuntu20.04
      resources:
        limits:
          nvidia.com/mig-1g.5gb: 1 # Requests a single 1g.5gb MIG device
  tolerations:
  - key: "nvidia.com/gpu"
    operator: "Exists"
    effect: "NoSchedule"

In the example above, the pod requests a specific MIG device type. This configuration allows the Kubernetes scheduler to assign the pod to a node with suitable MIG resources. This hardware-level isolation makes MIG the preferred solution for scenarios requiring security and performance consistency.

ℹ️ MIG Advantages

MIG provides hardware-level isolation, offering guaranteed performance and resource allocation for each GPU instance. This is critically important, especially for security-sensitive or latency-sensitive workloads.

GPU Time-Slicing Approach in Kubernetes

Time-Slicing is a software-level method for sharing a single physical GPU among multiple workloads. In this approach, the GPU's resources (compute units, memory) are allocated to different pods in time slices. This means the GPU processes only one task at a time, but by rapidly switching between tasks, it makes multiple tasks appear to run concurrently.

NVIDIA's nvidia-container-toolkit and Kubernetes device plugins enable this type of time-slicing capability. The device plugin exposes a GPU as multiple virtual GPUs, and each pod can request one of these virtual GPUs. Typically, pods make GPU resource requests using the standard nvidia.com/gpu request, but the device plugin's special configuration directs this request to time-slicing mode.

The biggest advantage of time-slicing is its flexibility. It can be applied to a wider range of GPUs (including older generation GPUs) than MIG and allows for finer-grained resource allocation. For example, you can share a GPU among multiple small inference pods; each pod receives an equal share of the GPU's processing time.

However, the main disadvantage of time-slicing is the lack of isolation. When one pod starts to fully utilize the GPU (the "noisy neighbor" effect), the performance of other pods sharing the same GPU can be negatively affected. For instance, a memory-intensive model can fill up the GPU memory, making memory allocation difficult for other pods, or a compute-intensive task can continuously occupy GPU cores, causing delays for other tasks. This situation can be unacceptable, especially for real-time or high-performance workloads.

apiVersion: v1
kind: Pod
metadata:
  name: time-sliced-pod
spec:
  restartPolicy: Never
  containers:
    - name: cuda-container
      image: nvcr.io/nvidia/cuda:11.4.0-runtime-ubuntu20.04
      resources:
        limits:
          nvidia.com/gpu: 1 # Requests a single virtual GPU
        requests:
          nvidia.com/gpu: 1
  tolerations:
  - key: "nvidia.com/gpu"
    operator: "Exists"
    effect: "NoSchedule"

While time-slicing enables more cost-effective and flexible use of GPUs, it lags behind MIG in terms of performance consistency and isolation guarantees. Therefore, the choice of approach must be carefully evaluated based on the specific requirements of the workloads.

MIG vs. Time-Slicing: Key Differences and Trade-offs

When choosing between MIG and Time-Slicing for GPU resource sharing, understanding the fundamental differences and associated trade-offs of both approaches is vital. This choice will directly impact the cost, performance, security, and operational complexity of your infrastructure.

Feature	NVIDIA MIG (Multi-Instance GPU)	Time-Slicing (Software-Based)
Isolation Level	Full hardware-level isolation (GPU instances are independent)	Software-level isolation (Shared GPU resources)
Performance	Guaranteed, consistent performance (no noisy neighbor effect)	Variable, performance susceptible to noisy neighbor effect
GPU Compatibility	Only specific NVIDIA GPUs with Ampere, Hopper, and Blackwell architectures	Wider range of GPUs (including older generation GPUs)
Resource Guarantee	Dedicated compute and memory resources	Time-slice based shared resources
Complexity	Requires initial configuration, then simple to operate	Configuration and monitoring can be more complex
Maximum Instances	Depends on GPU model (e.g., up to 7x 1g.5gb for A100)	Scalable to an unlimited number of pods (performance degrades)
Use Cases	Critical training, multi-tenant environments, high security	Lightweight inference, development/testing, batch processing, cost-focused

MIG offers hardware-guaranteed isolation and consistent performance but is only compatible with certain high-end GPUs, which can increase initial investment costs. Additionally, the number of MIG instances a GPU can be divided into is fixed, which can sometimes limit flexibility. For example, you can divide an A100 GPU into up to 7x 1g.5gb instances, but it might not be possible to create an 8th small workload MIG instance.

Time-slicing, on the other hand, offers a more cost-effective and widely compatible GPU option. It allows multiple small workloads to share the same GPU, maximizing GPU utilization. However, this flexibility comes with the risk of performance fluctuations and "noisy neighbor" issues. For example, a memory-intensive model can fill up the GPU memory, making memory allocation difficult for other pods, or a compute-intensive task can continuously occupy GPU cores, causing delays for other tasks. This situation can be unacceptable, especially for real-time or high-performance workloads.

⚠️ Important Trade-off: Isolation vs. Flexibility

MIG provides strong hardware-level isolation and performance guarantees, while Time-Slicing offers more flexibility and broader GPU compatibility. The choice depends on balancing your workload's isolation, performance consistency, and cost expectations.

Choosing the Right Approach: Scenarios and Decision Criteria

The choice of GPU sharing method in Kubernetes varies based on the nature of your workloads, security requirements, performance expectations, and budget constraints. An incorrect choice can lead to either unnecessary costs or performance issues.

Scenarios for Preferring NVIDIA MIG:

Security and Isolation Critical Workloads: Especially in multi-tenant environments or applications dealing with sensitive data (e.g., financial modeling on a bank's internal platform), hardware-level isolation is indispensable. MIG ensures each workload has its own dedicated GPU resources.
High Performance and Latency Guarantees: MIG is ideal for applications requiring performance consistency, such as real-time AI inference services or critical AI training jobs. The absence of noisy neighbor effects ensures predictable latency and throughput.
Enterprise Production Environments: For critical production workloads where SLAs are important, MIG offers a more reliable and manageable environment.
Large-Scale AI Training: Medium-sized training jobs that don't utilize an entire GPU but still require significant GPU resources can run more efficiently and in isolation with MIG.

Scenarios for Preferring Time-Slicing:

Cost-Focused and Lightweight Inference Workloads: For small AI models or bursty, low-intensity inference requests, Time-Slicing increases cost-effectiveness by filling the underutilized capacity of the GPU. For example, for AI-powered analytics running in the backend of my side product, Time-Slicing helped me utilize resources efficiently.
Development and Test Environments: In environments where developers need quick GPU access but performance guarantees are less critical, Time-Slicing offers a flexible resource pool.
Older Generation GPUs: If GPU models without MIG support are being used, Time-Slicing is the only way to utilize existing hardware more efficiently.
Batch Processing and Asynchronous Tasks: Batch jobs that are less sensitive to latency and only need to be completed within a certain timeframe can be easily managed with Time-Slicing. Performance fluctuations are generally acceptable for such workloads.

When making a decision, you should clearly understand your workload's GPU memory, processing power, and data transfer requirements. You also need to consider budget constraints and your existing hardware inventory.

Tips for Configuring GPU Sharing in Kubernetes

Enabling and managing GPU sharing in Kubernetes requires correct configuration steps and continuous monitoring. Whether you use MIG or Time-Slicing, there are fundamental steps and points to consider.

First, you need to ensure that your NVIDIA GPU drivers are correctly installed and that the NVIDIA Container Toolkit is running. This allows containers to access GPU resources.

# Check driver status
nvidia-smi

# Verify Docker uses nvidia-container-runtime (if Docker is used)
docker info | grep 'Runtimes'

Next, you should deploy the NVIDIA GPU Operator in your Kubernetes cluster. This operator detects GPUs on nodes and exposes them as custom resources to the Kubernetes API, also automatically managing drivers, container runtime, and the device plugin.

# Deploy NVIDIA GPU Operator with Helm (recommended method)
# Add Helm repo
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

# Install GPU Operator
# This command deploys NVIDIA drivers, container toolkit, and device plugin.
# It is recommended to review the Helm chart's values (values.yaml) before installation.
helm install --wait gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator --create-namespace

Configuration for MIG:
MIG enablement is typically done via the GPU operator or directly through node configuration. After enabling MIG on a node, you need to partition the GPU according to your desired MIG profiles.

⚠️ Dangerous Command Warning

The nvidia-smi commands below modify GPU hardware configuration and typically require sudo privileges. These operations may require stopping all processes using the GPU or rebooting the system and can lead to data loss. They should be used with caution in production environments, with backups made and dry-run/verification steps applied beforehand.

# List MIG devices on the node
nvidia-smi mig -lgi

# Enable MIG mode (usually done once)
# sudo nvidia-smi -i 0 -mig 1 # Enables MIG mode on GPU 0

# Set MIG profiles (example: partitioning an A100 into 3x 1g.5gb)
# These commands are usually managed by node startup scripts or the GPU operator.
# Profile IDs vary by GPU model. For A100, the 1g.5gb profile ID is 19.
# sudo nvidia-smi -i 0 -cgi 19,19,19 -C

When requesting MIG resources in your Kubernetes Pods, you should use the nvidia.com/mig-<size>.<memory> format, for example, nvidia.com/mig-1g.5gb: 1. The scheduler will find nodes with appropriate MIG instances to satisfy this request.

Configuration for Time-Slicing:
Time-Slicing is typically enabled with a special configuration of the NVIDIA Device Plugin. This is usually done via the nvidia-device-plugin's ConfigMap.

# Time-Slicing setting in NVIDIA Device Plugin ConfigMap (example)
apiVersion: v1
kind: ConfigMap
metadata:
  name: time-slicing-config # ConfigMap name
  namespace: gpu-operator # Namespace where GPU Operator is installed
data:
  config.yaml: |
    version: v1
    flags:
      migStrategy: none # Indicates that MIG is not used
    sharing:
      timeSlicing:
        enabled: true
        resources:
          - name: nvidia.com/gpu
            replicas: 4 # Exposes a single physical GPU as 4 virtual GPUs

After creating this ConfigMap, you may need to configure or restart the GPU Operator to use this ConfigMap. Pods can then access the time-sliced GPU by simply requesting nvidia.com/gpu: 1. However, it's important to carefully set CPU and memory limits for each pod to prevent performance issues.

💡 Monitoring and Optimization

In both approaches, regularly monitoring GPU usage (memory, compute, temperature) is critical. By collecting nvidia-smi output with tools like Prometheus and Grafana, you can identify potential performance bottlenecks or idle capacity. This data provides key indicators for evaluating the effectiveness of your GPU sharing strategy.

Conclusion

Managing GPU resources in a Kubernetes environment is a fundamental need for dynamic AI/ML workloads. NVIDIA MIG and Time-Slicing offer two powerful yet distinct solutions for this need. MIG is ideal for high-security or latency-sensitive applications that seek strong hardware-level isolation and performance guarantees. Especially if modern GPUs like A100, H100 have been invested in, MIG makes the most efficient use of this hardware's capacity by partitioning it.

On the other hand, Time-Slicing offers flexibility and cost-effectiveness across a wider range of GPUs. In scenarios where performance fluctuations are more acceptable, such as development, test environments, or lightweight inference workloads, Time-Slicing is a practical way to maximize existing GPUs. There have been instances where I preferred the Time-Slicing model for AI-powered optimization modules in a manufacturing company's ERP, considering the balance between cost sensitivity and flexibility.

Ultimately, making the right decision requires carefully evaluating your workload's specific requirements, performance expectations, security policies, and budget. Both approaches have their own advantages and disadvantages, and the "best" solution is always context-specific. When making this decision, continuously monitoring GPU usage and optimizing your infrastructure according to the evolving needs of your workloads are key to long-term success.