Deploying ML Models as Sidecar Containers

---
title: "ML Sidecars on Kubernetes: Cut Inference Costs 40–60% with MIG Partitioning and Triton gRPC"
published: true
description: "Deploy ML models as Kubernetes sidecar containers with MIG partitioning on A100s and Triton over gRPC to cut inference costs 40% without sacrificing latency."
tags: [kubernetes, cloud, performance, api]
canonical_url: https://blog.mvp-factory.dev/ml-sidecars-kubernetes-mig-partitioning
---

## What We Are Building

By the end of this workshop, you will have a working Kubernetes pod spec that co-locates a Triton Inference Server sidecar alongside your application container, shares GPU resources using NVIDIA MIG partitioning on A100s, and communicates over localhost gRPC — cutting inference infrastructure costs by 40–60% versus dedicated GPU nodes.

Let me show you a pattern I use in every production ML deployment.

---

## Prerequisites

- A Kubernetes cluster with at least one A100 GPU node
- NVIDIA device plugin and MIG Manager DaemonSet installed
- `kubectl` access and familiarity with pod specs
- A model repository accessible via S3 (or equivalent object storage)
- Basic familiarity with gRPC concepts

---

## Step 1: Understand Why Dedicated GPU Nodes Fail You

Most teams treat GPU nodes like CPU-bound services — one workload, one node. The result is GPU utilization averaging 20–35% across a fleet while the billing meter runs at 100%.

The breaking point comes when inference traffic is bursty and co-located with the application generating the requests. You pay for full GPU capacity to serve peak load, then burn money during troughs.

The sidecar pattern changes that calculus entirely.

---

## Step 2: Configure MIG Partitioning on Your A100

NVIDIA's Multi-Instance GPU (MIG) feature on A100s partitions a single physical GPU into up to seven isolated instances, each with dedicated memory and compute. Here is the profile table you will reference constantly:

| MIG Profile | GPU Memory | Compute % | Ideal For |
|---|---|---|---|
| `1g.10gb` | 10 GB | ~14% | Small classification models |
| `2g.20gb` | 20 GB | ~29% | Medium transformers |
| `3g.40gb` | 40 GB | ~43% | Large language models (7B) |
| `7g.80gb` | 80 GB | 100% | Full A100 allocation |

Use the NVIDIA MIG Manager DaemonSet — not manual `nvidia-smi mig` commands. It reconciles MIG configuration declaratively across your fleet and is far less error-prone at scale. The device plugin then surfaces MIG slices as extended Kubernetes resources, making bin-packing multiple inference sidecars onto a single A100 possible.

---

## Step 3: Write the Sidecar Pod Spec

Here is the minimal setup to get this working. The init container syncs your model from S3 before Triton starts, so the repository is populated before inference begins.

yaml
spec:
initContainers:
- name: model-sync
image: amazon/aws-cli:latest
command: ["aws", "s3", "sync", "s3://my-model-bucket/", "/models/"]
volumeMounts:
- name: model-store
mountPath: /models
containers:
- name: app
image: myapp:latest
resources:
requests:
cpu: "2"
memory: "4Gi"
- name: triton
image: nvcr.io/nvidia/tritonserver:24.01-py3
args: ["tritonserver", "--model-repository=/models", "--grpc-port=8001"]
resources:
requests:
nvidia.com/mig-2g.20gb: "1"
limits:
nvidia.com/mig-2g.20gb: "1"
volumeMounts:
- name: model-store
mountPath: /models
volumes:
- name: model-store
emptyDir: {}

Your application talks to Triton over `localhost:8001` gRPC — sub-millisecond transport latency, no service mesh overhead. The transport latency difference is real: gRPC over loopback consistently delivers p99 latencies under 1ms, versus 4–12ms for cross-node calls depending on your CNI.

---

## Step 4: Add Scheduling Resilience and Readiness Probes

Bin-packing alone is not enough. Use topology spread constraints to distribute pods across nodes:

yaml

Pod template labels

labels:
app: inference-sidecar

Topology spread on the pod spec

topologySpreadConstraints:

• maxSkew: 1 topologyKey: kubernetes.io/hostname whenUnsatisfiable: DoNotSchedule labelSelector: matchLabels: app: inference-sidecar

The docs do not mention this prominently, but model load times for 7B-parameter models routinely run 45–90 seconds. Your readiness probe `initialDelaySeconds` must account for this, or you will see silent inference failures at pod startup:

yaml
readinessProbe:
httpGet:
path: /v2/health/ready
port: 8000
initialDelaySeconds: 60
periodSeconds: 10
livenessProbe:
httpGet:
path: /v2/health/live
port: 8000
initialDelaySeconds: 30
periodSeconds: 10

---

## The Numbers: What This Actually Saves

Assuming a single A100 8x node at ~$8,200/month handling 2,000,000 inferences/month:

| Deployment Model | GPU Utilization | Effective Cost/Inference | vs. Dedicated |
|---|---|---|---|
| Dedicated GPU nodes | 22% avg | $0.0041 | baseline |
| Sidecar + MIG (3 slices) | 67% avg | $0.0025 | ~40% lower |
| Sidecar + MIG (7 slices) | 89% avg | $0.0016 | ~60% lower |

The 40% figure reflects a `2g.20gb` 3-slice configuration — the practical starting point for most teams.

---

## Gotchas

**Pod restarts take Triton with them.** When your application container crashes or rolls, Triton goes down too. For models with 60–90 second load times, that is a real availability hit during rolling updates and crash recovery. If uptime is the priority over cost, dedicated inference services decouple these lifecycles — that tradeoff is worth acknowledging before you commit.

**Host-level memory pressure is subtle.** MIG provides memory isolation between slices, but CPU OOM events and driver contention can degrade GPU throughput indirectly. Monitor `nvidia-smi dmon` at the node level, not just pod-level metrics. This one catches people off guard.

**Model size constraints are hard limits.** A `2g.20gb` slice caps you at 20GB GPU memory. A 13B parameter model in FP16 needs ~26GB — it will not fit. Profile your model footprint before committing to a MIG profile. TensorRT INT8 quantization is your primary lever when memory-constrained.

**Use gRPC, not HTTP, for intra-pod communication.** Protocol Buffers serialization overhead is negligible compared to transport latency gains over loopback. Generate the gRPC stubs and skip the REST layer entirely.

---

## Where to Start This Week

1. Audit GPU utilization before touching anything. If your nodes average below 50%, you are a candidate for MIG-based co-location. Pull CloudWatch or GKE GPU metrics and establish a real baseline first.
2. Start with `2g.20gb` MIG profiles. They fit most mid-size transformer workloads, give you three slices per A100, and skip the operational complexity of 7-slice configurations.
3. Wire Triton over gRPC on localhost from day one — retrofitting this later is annoying.

Long GPU optimization sessions call for regular breaks. I use [HealthyDesk](https://play.google.com/store/apps/details?id=com.healthydesk) for timed desk exercise reminders — a small habit that pays off during those 90-minute cluster debugging marathons.

**Relevant docs:**
- [NVIDIA MIG User Guide](https://docs.nvidia.com/datacenter/tesla/mig-user-guide/)
- [Triton Inference Server docs](https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/)
- [Kubernetes device plugin framework](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/)