Operational Playbook: Onboarding, Rolling Upgrades, and Fault Isolation

• Onboarding checklist: validations, resource quotas, and security
• Rolling upgrades that don't wake the pager (canaries, blue/green, migration)
• Crash containment: container limits, cgroups, and GPU isolation
• SRE playbook: incident response, postmortems, and continuous improvement
• Practical playbook: step-by-step checklists and runbook templates

Shared inference platforms buy you cost-efficiency and expose you to three unavoidable operational realities: bad tenants, risky upgrades, and resource contention. You stop the pager by making tenant onboarding, rolling upgrades, and fault isolation procedural, measurable, and automatable.

The symptoms you already recognize: a single tenant loads a few oversized models and pushes a node to memory pressure, Kubernetes evicts customer pods and the OOM killer restarts inference containers; a service mesh sidecar upgrade flips traffic and doubles latency for everyone; an upgrade without staged traffic causes a cascade of retries and CPU throttling. Those visible failures are rooted in weak onboarding gates, coarse upgrade practices, and missing hard isolation at the kernel and device level .

Onboarding checklist: validations, resource quotas, and security

What you validate on day one determines whether the tenant ever becomes a noisy neighbor.

• Validate the model artifact and runtime assumptions
  • Check model size, number of parameters, and peak memory per call. Record a baseline memory footprint and a cold and hot inference latency profile.
  • Run a short local perf test (perf_analyzer for Triton or a small load harness) and capture throughput at target p99 latency.
  • Confirm framework compatibility (TensorRT, PyTorch, ONNX runtime) and whether model initialization does heavy CPU/GPU work at load time (warm-up cost).
• Enforce resource contracts at admission
  • Require resources.requests and resources.limits on every Pod; enforce defaults with a LimitRange so tenants cannot create unbounded containers. LimitRange lets you set minimum/maximum request policies for CPU/memory per namespace.
  • Put a ResourceQuota per tenant namespace to cap aggregate CPU, memory, number of pods, and GPU counts (e.g., requests.nvidia.com/gpu). That prevents accidental cluster exhaustion.
• Gate security and supply-chain
  • Enforce image policies via admission webhooks: signed images, vulnerability scan status, and restricted registries. Use MutatingAdmissionWebhook to inject runtime decorators and ValidatingAdmissionWebhook to reject non-compliant specs.
  • Apply namespace-level RBAC, NetworkPolicy to isolate tenant traffic, and Pod Security admission (PSA) to enforce minimal privileges.
• Capacity and billing metadata
  • Onboard a metadata manifest containing expected RPS, SLA targets, and cost-center tags. That allows scheduling decisions (priority classes) and accurate chargeback.
• Automation checklist (what to run programmatically)
  • Static checks: model size, config.pbtxt sanity (for Triton), expected input/output shapes.
  • Dynamic checks: local perf profile, memory footprint, cold start time.
  • Admission: LimitRange + ResourceQuota + webhook validation as gates.

Example minimal ResourceQuota for a tenant namespace:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: tenant-a-quota
  namespace: tenant-a
spec:
  hard:
    requests.cpu: "16"
    requests.memory: "64Gi"
    limits.cpu: "32"
    limits.memory: "128Gi"
    requests.nvidia.com/gpu: "4"
    pods: "50"

Important: enforce both requests and limits (or use LimitRange defaults) so the scheduler has correct accounting and QoS classification works predictably. Kubernetes uses requests for scheduling and limits are enforced via the kernel (cgroups) — CPU is throttled, memory can lead to OOM kills.

Rolling upgrades that don't wake the pager (canaries, blue/green, migration)

Upgrades are the number-one source of multi-tenant pain. Treat them like controlled experiments.

• Canary deployments: weight-based traffic shifts
  • Use a traffic control plane (service mesh or gateway) to route a small percentage of traffic to the new model version and increase weight as metrics stay healthy. Istio’s weighted routing is a standard primitive for this.
  • Automate the analysis and promotion with a progressive delivery controller (Flagger, Argo Rollouts). Flagger integrates canaries with metrics (Prometheus) and will automatically roll back on regression.
• Blue/green when you need atomic cutovers
  • Blue/green works when model state and connection pinning make progressive increase undesirable. Keep a primary and canary service and switch the Service or VirtualService once the canary proves healthy.
• Rolling update knobs for Kubernetes Deployments
  • strategy.rollingUpdate.maxSurge and maxUnavailable tune risk vs speed. Pair with readinessProbe so new Pods only receive traffic when warm and healthy.
  • Respect PodDisruptionBudget to avoid reducing capacity under maintenance; define minimum availability for critical tenants.
• Verification signals you must include
  • Latency p99, error-rate, model output correctness (sampled golden inputs), and resource signals (GPU memory used, GPU SM utilization).
  • Use real-traffic canaries (small percent) rather than only synthetic testing for complex performance regressions.
• Migration considerations
  • When moving models between GPUs/nodes, observe memory-residency and GPU context setup times. For LLMs, cold loads can take seconds — require readiness gating until warm.

Example Deployment snippet (rolling update with readiness gating):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: model-service
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    spec:
      containers:
      - name: triton
        image: nvcr.io/nvidia/tritonserver:xx
        readinessProbe:
          httpGet:
            path: /v2/health/ready
            port: 8000
          initialDelaySeconds: 10
          periodSeconds: 5
        resources:
          requests:
            cpu: "2"
            memory: "8Gi"
          limits:
            cpu: "4"
            memory: "16Gi"

Compare at-a-glance:

Strategy	When to use	Pros	Cons
Rolling update	Stateless, low-risk changes	Fast, continuous	Hard to roll back traffic-level regressions
Canary (weight shift)	Perf-sensitive or correctness-sensitive	Incremental verification, safe rollback	Requires mesh/gateway and metrics
Blue/green	Atomic cutover or stateful migration	Quick rollback and clear stable version	Extra infra + potential double capacity cost

Cite the canary primitives and examples in Istio and Flagger for automation.

Crash containment: container limits, cgroups, and GPU isolation

When a tenant blows past limits you need hard walls at the OS and hardware layer.

• How requests vs limits behave in practice
  • requests drive scheduling and QoS classification; limits are enforced by the kubelet / runtime and ultimately by cgroups in the kernel. CPU gets throttled when it reaches CPU limits; memory exceedance can trigger the OOM killer and restart the container. Plan for that operationally.
• Use cgroups v2 features for stronger isolation
  • cgroups v2 exposes memory.max, memory.high, pids.max, and IO controls that let you throttle or hard-limit cross-tenant effects. The kernel cgroup v2 docs are the authoritative reference.
  • Example (host command to set hard memory cap for a cgroup): echo 8G > /sys/fs/cgroup/tenant-a.slice/memory.max (requires root and appropriate cgroup layout).
• Limit threads and file-descriptors
  • Enforce pids limits (pids.max) to stop runaway thread creation and nofile limits via the container runtime or sysctl.
• GPU isolation patterns
  • Use device-level isolation like NVIDIA MIG to carve GPUs into independent instances with dedicated compute and memory so tenants cannot evict each other at the device level. MIG gives you guaranteed fractional GPUs on supported hardware.
  • Alternatively, treat GPUs as extended resources (nvidia.com/gpu) and restrict allocation via ResourceQuota. For multi-model co-location on a GPU host, prefer Triton’s model control APIs so a single process can host many models without duplicative CUDA contexts. Triton supports explicit and poll-based model control modes to load/unload models at runtime.
• Kernel-level containment and OOM strategy
  • Tune oom_score_adj / OOM policy for critical system daemons, and ensure kubelet has eviction thresholds configured so node-level pressure triggers predictable pod evictions rather than random host instability. Kubernetes documents node eviction and memory QoS behaviors — use them to set expectations and probes.

Example Pod fragment that reserves a GPU and sets QoS toward Guaranteed (equal requests and limits):

spec:
  containers:
  - name: model
    image: myregistry/model:1.0
    resources:
      requests:
        cpu: "2000m"
        memory: "16Gi"
        nvidia.com/gpu: "1"
      limits:
        cpu: "2000m"
        memory: "16Gi"
        nvidia.com/gpu: "1"

Important: prefer Guaranteed QoS for latency-sensitive inference pods; Kubernetes will evict BestEffort then Burstable before Guaranteed under node pressure. Use cgroups v2 memory controls for fine-grained host-level behavior.

SRE playbook: incident response, postmortems, and continuous improvement

An SRE-grade platform turns incidents into disciplined learning loops.

• Alerting and runbooks
  • Attach a runbook_url (or runbook annotation) to every Prometheus alert so Alertmanager notifications carry direct remediation steps. The Prometheus alerting rule model supports annotations for runbook_url and action.
  • Example Prometheus rule fragment:

groups:
- name: inference.rules
  rules:
  - alert: TenantOOMsHigh
    expr: increase(kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}[5m]) > 0
    for: 2m
    labels:
      severity: page
    annotations:
      summary: "OOM kills detected for tenant {{ $labels.namespace }}"
      runbook_url: "https://internal.runbooks/tenant-ooms"
      action: "Check pod memory limits, review model load behavior, postmortem if repeated"

• Playbooks for first responders
  • Triage checklist (ordered, copyable into alert message):
  • Identify impacted tenant namespace and check kubectl get pods -n <tenant> and kubectl describe pod <pod> for OOMKilled.
  • Check node-level pressure: kubectl describe node <node> and kubelet eviction events.
  • Inspect GPU memory and processes: nvidia-smi -q -i <gpu> or DCGM metrics if available.
  • If immediate mitigation needed, scale down or pause the tenant’s Deployment or set kubectl patch to reduce replicas.
• Postmortems and learning
  • Adopt a blameless postmortem culture and document incidents with root cause, contributing factors, timeline, impact, and actionable fixes with owners and SLAs for completion. Google SRE and Atlassian provide pragmatic postmortem guidance and templates. Track remediation items to completion.
• Pager and escalation policy
  • Define clear pager thresholds: only page for sustained availability or safety issues. Route noisy (resource) alerts to an automation channel first so you can throttle noise and trigger human paging only when automation fails.
• Continuous improvement
  • Use postmortem metadata to track incident classes (e.g., OOM, upgrade regression, hardware failure) and reduce recurrence through automation, better onboarding gates, or targeted quotas.

Important: put the actionable remediation (commands and a short checklist) into the alert payload via annotations.runbook_url so the on-call engineer can act in seconds rather than minutes.

Practical playbook: step-by-step checklists and runbook templates

Below are immediately usable checklists and templates you can drop into your platform ops repo.

Onboarding checklist (apply before tenant receives production traffic)

1. Automated static checks
   • Model size < X GB, accepted format, config sanity
   • Image signed and vulnerability scan passes policy
2. Resource contract
   • Create namespace tenant-x
   • Apply LimitRange defaults and ResourceQuota (CPU, memory, GPU)
3. Performance validation
   • Run perf_analyzer or small load test to capture p50/p95/p99, cold start, memory footprint
4. Deploy to canary (1 replica), route 1–5% traffic
   • Attach alerting rules for latency and error-rate
5. Approve to roll to production only if metrics pass for X minutes

Rolling upgrade runbook (short)

1. Start canary (create canary Deployment or new revision)
2. Warm model: ensure readinessProbe returns success after warm-up
3. Monitor: sample outputs, check p99, GPU memory, and success rate
4. Increment traffic weight: 5% → 25% → 50% → 100% with checks between steps (use Flagger/Argo)
5. If regression: immediate rollback and mark the deployment as failed for analysis

Incident triage runbook (first 10 minutes)

1. Confirm alert and scope (kubectl get pods -A | grep <tenant>).
2. Check Pod status and events: kubectl describe pod -n <ns> <pod> — look for OOMKilled.
3. Check node metrics and eviction events: kubectl describe node <node>.
4. Check GPU status: kubectl exec -n kube-system -it <gpu-tooling-pod> -- nvidia-smi (or DCGM dashboards).
5. If tenant caused resource exhaustion: scale down their replicas or kubectl cordon/evict as temporary isolation.
6. Post-incident: open a postmortem ticket, assign owner, and schedule remediation with SLO.

Runbook snippet — basic commands

# List pods and status for tenant
kubectl get pods -n tenant-a -o wide

# Check recent terminations
kubectl get events -n tenant-a --sort-by='.lastTimestamp' | tail -n 50

# Describe a problematic pod
kubectl describe pod -n tenant-a model-12345

# Check node resource pressure
kubectl describe node <node-name>

# Inspect GPU usage (on node)
ssh operator@<node>
nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu --format=csv

Important: convert recurring fixes into automation (e.g., automatic canary rollback, automatic tenant throughput throttling) and measure the reduction in pages and MTTR.

Sources:
Resource Management for Pods and Containers - Kubernetes documentation on requests, limits, how CPU is throttled and memory can cause OOMs; guidance for resource units and examples.

Pod Quality of Service Classes - Kubernetes doc describing QoS classes (Guaranteed, Burstable, BestEffort) and eviction behavior.

Resource Quotas - Kubernetes documentation describing ResourceQuota usage, including quota for requests.nvidia.com/gpu and quota scopes.

Limit Ranges - Kubernetes concept page for LimitRange to enforce per-namespace defaults and min/max constraints.

Admission Control in Kubernetes - Kubernetes admission controllers, including MutatingAdmissionWebhook and ValidatingAdmissionWebhook.

Control Group v2 — The Linux Kernel documentation - Authoritative kernel documentation on cgroup v2 features (memory.max, memory.high, pids.max) and behaviors.

MIG User Guide — NVIDIA Multi-Instance GPU - NVIDIA guide describing MIG partitions and how they provide dedicated compute/memory slices for multi-tenant isolation.

Model Management — NVIDIA Triton Inference Server - Documentation on Triton’s model control modes (NONE, POLL, EXPLICIT) and load/unload semantics.

Flagger — progressive delivery for Kubernetes - Flagger docs showing automated canary promotion based on metrics, integrations and examples.

Specifying a Disruption Budget for your Application (PodDisruptionBudget) - Kubernetes guide on how to use PodDisruptionBudget to limit concurrent disruptions during rollouts.

Alerting rules | Prometheus - Prometheus rules reference describing labels and annotations (used to attach runbook_url and actionable guidance to alerts).

Rate limit — Envoy documentation - Envoy documentation on local and global rate limiting filters, useful for protecting the platform from traffic spikes.

Postmortem Culture: Learning from Failure - Google SRE guidance on blameless postmortems, storing and tracking action items, and cultural practices for continuous learning.

Incident postmortems (Atlassian) - Atlassian’s postmortem handbook describing templates, approvers, and improvements tracking.