DEV Community: Harshit Luthra

Why your Docker build caches locally and never in CI

Harshit Luthra — Fri, 25 Sep 2026 12:47:03 +0000

Originally published at harshit.cloud on 2026-09-02.

The build is quick on your laptop and slow in CI, and the Dockerfile is identical. This is not a mystery about Docker. The layer cache lives in the local layer store, and a CI runner is a clean machine with an empty one. Every CI build is a first build.

Fixing it is two separate jobs. Make the cache work at all, which is about instruction order and what enters the build context, and then make the cache survive the runner, which is about exporting it somewhere that outlives the job. The first job also speeds up your laptop. The second only matters in CI.

what actually invalidates a layer

BuildKit keys each instruction on its inputs. For RUN, the input is the command string plus the state of everything before it. For COPY and ADD, the input is the contents and metadata of the files being copied.

The part that catches people is that invalidation cascades. Once one instruction misses, every instruction after it rebuilds, whether or not its own inputs changed. So a Dockerfile that copies the whole source tree before installing dependencies rebuilds the dependency install on every commit, forever:

FROM node:22-slim
WORKDIR /app
COPY . .                  # any source change invalidates here
RUN npm ci                # ...so this always reruns
CMD ["node", "server.js"]

The fix is old and still the highest-value change in most Dockerfiles. Copy only the files the install reads, install, then copy the rest:

FROM node:22-slim
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci                # only reruns when the lockfile changes
COPY . .
CMD ["node", "server.js"]

Same instructions, different order, and the install stops rerunning for a one-line change in a route handler. The rule generalises to every ecosystem: go.mod and go.sum before the source, requirements.txt before the package, Cargo.toml and a dummy main.rs before the crate.

the .dockerignore that decides whether any of this works

COPY . . hashes the build context. If the context contains .git, the hash changes on every commit, including commits that touch nothing the image needs. If it contains a local node_modules, the hash changes whenever you install anything on your laptop, and the image you build is not the image CI builds.

.git
node_modules
.next
dist
coverage
*.log
.env*
Dockerfile
.dockerignore

Two things this buys beyond cache hits. The context stops being uploaded to the daemon, which on a repo with a large .git is most of the wall-clock time before the build even starts. And .env* stops being copied into a layer where anyone with the image can read it, which is a different post's problem but a real one.

You can see the size of what you are sending in the first line of build output:

$ docker build .
[+] Building 0.4s (8/8) FINISHED
 => [internal] load build definition from Dockerfile          0.0s
 => [internal] load .dockerignore                             0.0s
 => => transferring context: 2.31kB                           0.0s

transferring context: 2.31kB is a healthy number. If it says 340MB, the .dockerignore is missing or wrong. There is a shorter version of this specific trap in Docker build cache: the .dockerignore gotcha.

cache mounts, for the work that repeats

Instruction ordering stops the install rerunning when nothing changed. It does nothing for the case where the lockfile genuinely did change and you would still rather not download every package again.

That is what cache mounts are for. RUN --mount=type=cache mounts a persistent directory into one step, and the directory never becomes part of the layer:

# syntax=docker/dockerfile:1
FROM node:22-slim
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm \
    npm ci
COPY . .
CMD ["node", "server.js"]

The # syntax= line on the first line is required. Without it, the Dockerfile is parsed by the built-in frontend, which does not understand --mount, and you get a syntax error that reads like the flag does not exist.

The same shape for the other ecosystems:

RUN --mount=type=cache,target=/root/.cache/pip pip install -r requirements.txt
RUN --mount=type=cache,target=/go/pkg/mod go build ./...
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
    apt-get update && apt-get install -y --no-install-recommends curl

The apt one needs sharing=locked because two concurrent builds writing to the same apt cache will corrupt it. locked serialises them. The default, shared, is right for package managers that handle concurrent readers themselves, which npm, pip, and the Go module cache all do.

Cache mounts also fix the pattern where people delete the cache to keep the layer small, which is a real cost you no longer have to pay:

# no longer necessary: the cache never entered the layer
RUN apt-get install -y curl && rm -rf /var/lib/apt/lists/*

making the cache outlive the runner

Everything above still produces a cold build in CI, because none of it survives the machine. BuildKit can export the cache to a registry or to the CI provider's own cache store, and import it at the start of the next run.

For GitHub Actions, the provider-native backend is the least work:

- uses: docker/setup-buildx-action@v3
- uses: docker/build-push-action@v6
  with:
    push: true
    tags: ghcr.io/example/api:${{ github.sha }}
    cache-from: type=gha
    cache-to: type=gha,mode=max

mode=max is the part people leave off and then wonder why a multi-stage build is still slow. The default, min, exports only the layers that end up in the final image. In a multi-stage Dockerfile the expensive stage is the builder, and none of it is in the final image, so min caches precisely the cheap half.

The GitHub Actions cache is capped at 10 GB per repository and evicts least-recently-used entries, so a repo with several images can quietly evict its own cache between runs. If that happens, or if you build outside GitHub, export to the registry instead:

    cache-from: type=registry,ref=ghcr.io/example/api:buildcache
    cache-to: type=registry,ref=ghcr.io/example/api:buildcache,mode=max

That writes the cache as a separate tag next to the image. It costs registry storage and it does not evict on you.

proving it worked

The reason to check rather than assume is that a cache configuration can be entirely valid and still miss everything, and the build output looks the same either way unless you ask for it.

docker buildx build --progress=plain --cache-from type=registry,ref=ghcr.io/example/api:buildcache .

--progress=plain prints every step with its status instead of collapsing them:

#8 [3/5] RUN --mount=type=cache,target=/root/.npm npm ci
#8 CACHED

#9 [4/5] COPY . .
#9 DONE 0.2s

CACHED on the install step and DONE on the copy is the correct shape: the dependency layer was reused, the source layer was rebuilt. If the install step says DONE 47.3s on a run where the lockfile did not change, something above it invalidated, and the step immediately before it is where to look.

Key Insight: Cache invalidation cascades forward and never backward. When a step misses unexpectedly, the bug is always in a step above it, never in the step that reported the miss. Read the build output from the top down and stop at the first thing that rebuilt.

the one that will still catch you

ARG values participate in the cache from the point they are used. A build argument that changes every run, and CI is full of them, invalidates everything downstream of its first reference:

ARG GIT_SHA
RUN echo "$GIT_SHA" > /app/version     # invalidates on every commit
COPY package.json package-lock.json ./
RUN npm ci                             # ...and so does this

Move it as late as possible. The version stamp belongs after the expensive work, not before it:

COPY package.json package-lock.json ./
RUN npm ci
COPY . .
ARG GIT_SHA
RUN echo "$GIT_SHA" > /app/version     # invalidates only itself

The same applies to any ENV set from a build argument, and to LABEL values containing a timestamp, which is a popular way to make a Dockerfile uncacheable while believing you are adding metadata.

Check what you actually have with docker buildx du, which shows what the cache is holding and how much of it is reclaimable. If the number never grows between CI runs, the export is not working and no amount of Dockerfile tuning will help.

More on the container side of things under Docker and containers, the pipeline side under ci-cd. If your builds are fast and your images are still running as root, Docker security: stop running everything as root is the other half of the same file. And if the reason you are reading this is that the CI bill went up rather than the builds got slow, AWS cost optimization covers where that money usually actually goes.

Start with the .dockerignore. It takes two minutes, it is the one change that helps locally and in CI at the same time, and a repo without one almost always has a COPY . . that has not hit the cache in months.

Alert on the error budget, not the CPU graph

Harshit Luthra — Fri, 25 Sep 2026 12:46:23 +0000

Originally published at harshit.cloud on 2026-08-21.

The alert I have silenced the most times in my life is HighCPUOnNode. It fires, someone opens the dashboard, CPU is at 85%, nothing is wrong, and the alert gets acknowledged and forgotten. Six months of that and the whole channel is furniture.

The problem is not the threshold. It is that CPU is a cause, and nobody is paid to care about causes at 3am. What matters is whether the service is failing its users fast enough to matter. Burn-rate alerting measures exactly that: how quickly you are spending the error budget the SLO gives you, with a short second window so the page stops when the incident does.

the budget, in one line of PromQL

An SLO of 99.9% availability over 30 days gives you a 0.1% error budget. That is the entire quantity being managed. Every alert below is a statement about the rate at which it is being spent.

The ratio itself is the thing you want precomputed, because you will query it over several windows and Prometheus should not recompute it per alert evaluation:

groups:
  - name: slo-api
    interval: 30s
    rules:
      - record: job:slo_errors_per_request:ratio_rate5m
        expr: |
          sum(rate(http_requests_total{job="api",code=~"5.."}[5m]))
            /
          sum(rate(http_requests_total{job="api"}[5m]))
      - record: job:slo_errors_per_request:ratio_rate1h
        expr: |
          sum(rate(http_requests_total{job="api",code=~"5.."}[1h]))
            /
          sum(rate(http_requests_total{job="api"}[1h]))

Repeat for 30m, 6h, 6h, and 3d. It is repetitive and it is worth it: the alert rules that follow are then one comparison each, and the windows are stated in one place where you can audit them.

Note the numerator. code=~"5.." counts server errors. It does not count 4xx, because a client sending malformed requests is not the service failing, and the fastest way to make an SLO meaningless is to let someone else's bad client drain your budget.

burn rate is just the ratio divided by the budget

If your budget is 0.1% and you are currently serving 1.44% errors, you are burning at 14.4 times the sustainable rate. That is the whole calculation.

The useful thing about that number is what it implies about time. Burn rate 1 exhausts the budget exactly at the end of the 30-day window. Burn rate 14.4 exhausts it in roughly 50 hours, which means it eats 2% of the month's budget in a single hour. That is the threshold worth waking someone for.

Budget spent	Over	Burn rate	Long / short window	Response
2%	1 hour	14.4	1h / 5m	page
5%	6 hours	6	6h / 30m	page
10%	3 days	1	3d / 6h	ticket

Those three rows are the multi-window setup from the Google SRE Workbook's alerting chapter, and they are a better starting point than anything you will derive from scratch. The first row catches a hard outage in minutes. The second catches the partial degradation that a 1-hour window would take too long to notice. The third catches a slow leak that nobody should be woken for.

the short window is what makes it usable

The single-window version of this alert has a well-known failure: it keeps firing long after the problem is gone. A total outage lasting five minutes pushes the 1-hour error ratio above the threshold, and it stays above for close to an hour afterwards while the bad five minutes ages out of the window. The pager goes off, you fix it in four minutes, and it keeps going off.

The fix is to require both windows to be over the threshold at once:

- alert: ErrorBudgetBurnFast
  expr: |
    (
      job:slo_errors_per_request:ratio_rate1h{job="api"} > (14.4 * 0.001)
      and
      job:slo_errors_per_request:ratio_rate5m{job="api"} > (14.4 * 0.001)
    )
  labels:
    severity: page
  annotations:
    summary: "api is burning error budget 14.4x (2% of the month in an hour)"

- alert: ErrorBudgetBurnSlow
  expr: |
    (
      job:slo_errors_per_request:ratio_rate6h{job="api"} > (6 * 0.001)
      and
      job:slo_errors_per_request:ratio_rate30m{job="api"} > (6 * 0.001)
    )
  labels:
    severity: page
  annotations:
    summary: "api is burning error budget 6x (5% of the month in six hours)"

The long window says the incident is big enough to page for. The short window says it is still going. Convention is short = long / 12, which is where 5m/1h and 30m/6h come from.

This also lets you drop the for: clause, which is the other thing people reach for and should not. for: requires the expression to be continuously true for its whole duration, and a single evaluation where the value dips below the threshold resets the timer. During a flapping incident, for: 10m can fail to fire for an hour. The short window gives you the same "is this still real" check without the reset behaviour.

Key Insight: A single-window burn-rate alert will keep paging for up to an hour after the incident is over, because the bad minutes are still inside the window. The short second window is not a refinement. It is the thing that makes the alert trustworthy enough to leave enabled.

the alert that fires when nothing fires

Every ratio-based alert has the same hole. If the scrape target disappears, http_requests_total stops producing samples, the division produces no series, and the alert quietly evaluates to nothing. Total outage, silent pager.

Two lines close it:

- alert: ApiMetricsAbsent
  expr: absent_over_time(http_requests_total{job="api"}[10m])
  labels:
    severity: page
  annotations:
    summary: "no request metrics from api for 10 minutes"

absent_over_time returns 1 when the selector has matched nothing for the whole range, which covers both a dead target and a metric that got renamed in a refactor. Use absent_over_time rather than absent, because absent fires on a single missed scrape and will page you for a rolling deploy.

Verify it before you trust it, by asking Prometheus what the expression returns right now:

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=absent_over_time(http_requests_total{job="api"}[10m])' | jq '.data.result'

[]

Empty is correct while the target is healthy. If you get a result with "value": [..., "1"] and the service is up, your label selector does not match the series you think it does, which is worth finding out now rather than during an incident.

what happens to the cause-based alerts

They stay. They just stop paging.

CPU, memory, disk, queue depth, replica count: all of it remains useful, and all of it is what you look at ten seconds after the page arrives. The change is the routing. A symptom alert wakes someone. A cause alert opens a ticket or sits on a dashboard until a human is already looking.

There is a good reason to keep a small number of cause-based pages, and it is prediction rather than diagnosis. Disk filling at a rate that hits 100% in four hours is worth a page, because by the time it becomes a symptom the recovery is much more expensive:

- alert: DiskWillFill
  expr: predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 4 * 3600) < 0
  for: 30m
  labels:
    severity: page

That is the exception, and for: 30m is appropriate here precisely because predict_linear is noisy and you do want it to settle.

The rest of the cause alerts drop to ticket severity, and Alertmanager stops sending three pages for one incident:

inhibit_rules:
  - source_matchers: [severity = "page"]
    target_matchers: [severity = "ticket"]
    equal: [job]

the failure mode this creates

Being honest about the trade: SLO-based alerting moves the argument from "is this threshold right" to "is this SLO right", and the second argument is harder and more political. A 99.9% target that nobody agreed to is a number you will end up negotiating during an incident, which is the worst possible time.

It also means a slow, permanent degradation that stays under burn rate 1 never pages at all. That is intentional, and it will still feel wrong the first time a latency regression rides along for three weeks underneath the alerting threshold. The answer is the third row of the table, the ticket-level alert at burn rate 1, and actually reading the tickets.

The other honest caveat: this only measures what the SLI counts. An availability SLI built on 5xx will not notice a service returning 200 with an empty body, which is a real outage that looks perfect on the graph. How I took down 30% of production with one TLS fingerprinting rule is the version of that where the requests never reach the application to be counted at all.

where to start

If you have one service and no SLO, the smallest useful version is one SLI, one target, two page alerts and one absent alert. That is roughly forty lines of YAML and it will replace most of a threshold-alert file.

The stack this sits on, including the recording-rule layout and the Grafana side, is in Prometheus and Grafana: from zero to production monitoring. The equivalent question for LLM serving, where GPU utilisation is the HighCPUOnNode of that world and queue time is the metric that actually predicts a bad experience, is in what a green GPU dashboard hides. If the cost of keeping all these series is the thing standing in your way, VictoriaLogs vs Loki benchmarks the logs half of the same problem. More of both under monitoring and observability.

Delete HighCPUOnNode on the way out. If it has never once been the reason someone found an incident, it was never an alert. It was a metric with a pager attached.

CrashLoopBackOff: five causes and how to tell them apart

Harshit Luthra — Fri, 25 Sep 2026 12:46:17 +0000

Originally published at harshit.cloud on 2026-08-12.

A pod in CrashLoopBackOff tells you almost nothing. It is the waiting state, not the failure: the container exited, and the kubelet is sitting out a back-off before trying again. The failure happened one restart ago and is recorded somewhere else.

Two commands narrow it to one of five causes before you open the application logs. kubectl describe pod gives you the last termination reason and exit code, and kubectl logs --previous gives you what the dead instance printed. Everything below is a way of reading those two outputs.

the two commands, in order

Start with describe, because the exit code does most of the classification work:

kubectl describe pod api-7f4b9c6d8f-9m2tq

The part worth reading is not the top. It is the Last State block and the events at the bottom:

    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Tue, 12 Aug 2026 09:14:22 +0530
      Finished:     Tue, 12 Aug 2026 09:14:51 +0530
    Restart Count:  6

Reason and Exit Code are the diagnosis. Started and Finished are the second most useful pair in the block, because the gap between them separates "died instantly" from "ran for a while and then died", and those are different bugs.

Then read what the dead instance said:

kubectl logs api-7f4b9c6d8f-9m2tq --previous

Without --previous you get the current instance, which in a crash loop has usually just started and printed nothing. This is the single most common reason people conclude a crashing pod has no logs.

exit code 1 or 2: the application decided to quit

An exit code of 1 with output in the previous log is the easy case. The process started, hit something it did not like, and exited on purpose. Missing environment variable, a config file it could not parse, a database it could not reach on startup.

$ kubectl logs api-7f4b9c6d8f-9m2tq --previous
2026-08-12T09:14:22Z FATAL config: DATABASE_URL is required

The reason this one is worth naming separately is that it is the only cause where the application log is the answer. For the other four, the log is either empty or misleading, and you need the pod's own state instead.

If the variable is supposed to come from a Secret or ConfigMap, check that it actually arrived rather than trusting the manifest:

kubectl get pod api-7f4b9c6d8f-9m2tq -o jsonpath='{.spec.containers[0].env}' | jq

There is more on pulling exactly the field you want out of an object in kubectl JSONPath: extract exactly what you need.

exit code 137 with OOMKilled: the memory limit

137 is 128 + 9, so the process took a SIGKILL. If Reason says OOMKilled, the kernel's OOM killer did it because the container exceeded its memory limit.

The tell that separates a genuine limit problem from a memory leak is the gap between Started and Finished. Killed within seconds every time means the limit is below what the process needs at startup, usually a JVM heap or a model load. Killed after ten minutes, then twenty, then five, means it is leaking or the workload is spiky.

kubectl get pod api-7f4b9c6d8f-9m2tq -o jsonpath='{.spec.containers[0].resources}'

{"limits":{"memory":"256Mi"},"requests":{"cpu":"100m","memory":"128Mi"}}

Raising the limit is the fix for the first case and a delay for the second. What matters more than the number is that requests and limits are both set: a container with a limit and no request gets scheduled onto a node that cannot actually give it that memory, and you meet the OOM killer under load rather than at startup.

Key Insight: Exit code 137 does not always mean out of memory. It means SIGKILL. The kubelet sends the same signal when a liveness probe fails. Read the Reason field next to the exit code, not the exit code alone.

exit code 137 with no OOMKilled: the probe shot it

Same signal, different killer. If the last state reason is Error rather than OOMKilled, and there is a probe failure in the events just before the kill, then the liveness probe ended a container that was working.

Events:
  Type     Reason     Age                From     Message
  ----     ------     ----               ----     -------
  Warning  Unhealthy  2m (x9 over 8m)    kubelet  Liveness probe failed: Get "http://10.0.3.14:8080/health": dial tcp 10.0.3.14:8080: connect: connection refused
  Normal   Killing    2m (x3 over 8m)    kubelet  Container api failed liveness probe, will be restarted

connection refused during startup means the probe arrived before the process was listening. The container gets killed, restarts, and never survives long enough to answer, so the loop is self-sustaining. The fix is a startupProbe, which suspends liveness and readiness entirely until it passes:

startupProbe:
  httpGet: { path: /health, port: 8080 }
  periodSeconds: 5
  failureThreshold: 30      # 30 x 5s = 150s of startup budget
livenessProbe:
  httpGet: { path: /health, port: 8080 }
  periodSeconds: 10
  failureThreshold: 3       # after startup, a real hang is caught in 30s

Raising initialDelaySeconds on the liveness probe instead looks like the same fix and is not. It blinds you to genuine deadlocks for the whole delay, on every restart, forever. The startup probe buys the same time and then hands liveness back its tight interval. Your model isn't crashing, your probe is walks through the version of this that eats an afternoon, where the process is a model server and the startup budget is ten minutes rather than two.

exit code 127, 126, or a StartError: the process never ran

An empty --previous log with a non-zero exit is a different class of problem. Nothing ran, so nothing logged.

127 is command not found. The entrypoint or the command: in the manifest points at a path that is not in the image.
126 is found but not executable. Usually a script without the execute bit, or a shell script with a CRLF line ending so the kernel looks for an interpreter named /bin/sh\r.
StartError or CreateContainerError in Reason means the kubelet could not launch the process at all, and the events say why.

The events are more specific than the exit code here, so read them:

  Warning  Failed  30s (x4 over 90s)  kubelet  Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: exec: "/app/server": stat /app/server: no such file or directory

That is a build problem wearing a runtime costume. The image does not contain the binary the manifest asks for. Checking the image directly beats re-reading the Dockerfile:

docker run --rm --entrypoint ls ghcr.io/example/api:2.4.1 -la /app

A related trap is a volume mounted over the directory holding the binary, which produces the same error from a perfectly good image. Docker volume debugging: finding where your data actually lives covers pinning down what is actually at a mount point.

exit code 0: it finished, and Kubernetes disagreed

The strange one. Exit code 0 means the process completed successfully, and the pod is still crash-looping, because a Deployment's pods carry restartPolicy: Always and Kubernetes restarts a successful exit exactly as eagerly as a failed one.

This is nearly always a workload in the wrong object. A migration script, a backfill, a report generator: something that is supposed to run once and stop. It belongs in a Job, where restartPolicy: OnFailure or Never is available and completion is a terminal state rather than an invitation:

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate
spec:
  backoffLimit: 3
  template:
    spec:
      restartPolicy: OnFailure
      containers:
        - name: migrate
          image: ghcr.io/example/api:2.4.1
          command: ["/app/migrate"]

The other version of exit code 0 is a long-running server whose main process backgrounds itself and lets PID 1 return. The container is doing exactly what you told it to. It is just that what you told it was "start the thing and exit".

the sixty-second triage

Everything above collapses into one pass over the pod's state. Get the reason and exit code for every container in one shot rather than reading a screen of describe output:

kubectl get pod api-7f4b9c6d8f-9m2tq \
  -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.lastState.terminated.reason}{"\t"}{.lastState.terminated.exitCode}{"\n"}{end}'

api OOMKilled   137

Then branch on what comes back:

Exit code 1 or 2, log has output. Read the log. It is an application error.
Exit code 137, reason OOMKilled. Memory limit. Check the gap between Started and Finished to tell a too-small limit from a leak.
Exit code 137, reason Error, probe failure in the events. The liveness probe killed a healthy container. Add a startupProbe.
Exit code 126, 127, or a StartError. The process never launched. Read the events, then inspect the image.
Exit code 0. Wrong workload type. This wants to be a Job.

The reason to run this before opening the application logs is that three of the five causes leave no application logs at all, and one of them leaves logs that look fine right up to the moment something external kills the process. The pod's own state is the more honest witness.

If none of the five fit, the next thing to check is whether the container is being killed before it is scheduled at all, which is a different failure that presents as Pending rather than CrashLoopBackOff. That one, along with the networking failures that dress up as application bugs, is in five Kubernetes debugging tricks that saved my production. The rest of the Kubernetes writing here is collected under the Kubernetes tag, and the wider set of symptom-versus-cause posts under debugging.

The habit worth keeping is smaller than any of this. Before reading a single line of application output, run describe and write down two things: the exit code and the reason. Most crash loops stop being mysterious right there.

Two tenants, one GPU, and no wall between them

Harshit Luthra — Thu, 02 Jul 2026 19:15:37 +0000

Originally published at harshit.cloud on 2026-07-25.

Run nvidia-smi inside a pod in one namespace, then inside a pod in another, on a cluster using GPU time-slicing. Look at the GPU UUID and the PCI bus ID.

This is the last part. Seven parts built the fleet, wired it, scaled it, watched it, routed to it, and kept it alive through deploys. This one is about the moment more than one team shares it, which is where the assumptions that held for a single tenant quietly stop being true. The load-bearing one, the thing most teams get wrong: a Kubernetes namespace is not a wall on the GPU.

the isolation you don't have

A namespace is an API-scoping and RBAC boundary. It has nothing to do with the hardware. The device plugin advertises nvidia.com/gpu resources and the scheduler places pods onto them regardless of which namespace asked, so "one GPU, four replicas" under time-slicing means the scheduler can land a pod from team-a and a pod from team-b on the same physical die with no memory partition between them. What isolation you actually get depends entirely on the sharing mechanism, and it's worth stating exactly:

Fig. 2 · the only two rows with a real memory wall are "whole GPU" and "MIG." MPS caps are software. Time-slicing has nothing. If your mental model was "different namespace, different memory," this is the row that corrects it.

The config that creates this situation is ordinary and common. This is a device-plugin time-slicing setup, the kind people turn on to raise utilization:

sharing:
  timeSlicing:
    resources:
    - name: nvidia.com/gpu
      replicas: 4          # one physical GPU -> four schedulable "GPUs", zero isolation

MPS swaps that block for per-client soft caps (CUDA_MPS_PINNED_DEVICE_MEM_LIMIT, a compute percentage), which are enforced by the driver, not by hardware, and share a failure domain. This isn't only a noisy-neighbor concern. Security researchers have shown covert and side channels through the GPU's shared uncore engines, readable with unprivileged NVML calls, that bypass both MPS and MIG partitioning, and GPU DRAM that isn't zeroed on context teardown has leaked data across tenants under time-slicing. The one-line defensive setting: turn on renameByDefault: true so a shared GPU advertises as nvidia.com/gpu.shared, and a tenant can't request a "shared" GPU thinking it's a private one.

Key Insight: A Kubernetes namespace isolates the API and RBAC, never GPU memory. Under time-slicing or MPS, two tenants can sit on the same physical die with no hardware wall between their VRAM. If tenants don't trust each other, only MIG or a whole-GPU allocation is safe. This is the correction most GPU-cluster security models are missing.

MIG when tenants can't trust each other

MIG is the answer when the boundary has to be real. It carves the physical card into hardware partitions, each with its own DRAM slice, L2, and SMs, so one instance cannot read another's memory regardless of driver or kernel bugs. Expose it through the GPU Operator in mixed strategy, which advertises each profile as its own resource:

kubectl patch clusterpolicies.nvidia.com/cluster-policy --type='json' \
  -p='[{"op":"replace","path":"/spec/mig/strategy","value":"mixed"}]'

The node then advertises MIG profiles as first-class schedulable resources, and a tenant requests a hardware-isolated slice by name:

resources:
  limits:
    nvidia.com/mig-1g.10gb: 1     # one hardware-isolated 10GB partition

The tradeoff is the one from part 1: partitions are fixed at configure time, reconfiguring drains the node's GPU pods, and a workload can't burst past its slice. That rigidity is the price of a real wall. For untrusted multi-tenancy it's a price worth paying; for a single trusted team it's usually not.

quotas that don't waste the cluster

Isolation stops one tenant from reading another's memory. Quotas stop one tenant from eating the whole cluster. The blunt version is a ResourceQuota capping GPU requests per namespace (extended resources are quotable only via requests.):

apiVersion: v1
kind: ResourceQuota
metadata: { name: team-a-gpu-quota, namespace: team-a }
spec:
  hard:
    requests.nvidia.com/gpu: "8"

That's static, though, and it strands GPUs whenever a team is idle. Kueue fixes that by admitting jobs against quota and letting teams in a shared cohort borrow each other's idle GPUs while guaranteeing each its floor back on demand. A ClusterQueue holds the quota; the cohortName is what enables borrowing:

apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata: { name: team-a-cq }
spec:
  cohortName: "gpu-pool"                 # teams in one cohort lend/borrow idle GPUs
  resourceGroups:
  - coveredResources: ["nvidia.com/gpu"]
    flavors:
    - name: h100-flavor
      resources:
      - name: nvidia.com/gpu
        nominalQuota: 8                  # guaranteed floor
        borrowingLimit: 4                # cap on what it can borrow from the cohort

One version trap worth checking before you copy that: current Kueue serves v1beta2, where cohorts are a named field (and can be their own CRD with fair-share weights). Plenty of clusters still run v1beta1, where the field is a plain spec.cohort string and the standalone Cohort doesn't exist. kubectl get crd clusterqueues.kueue.x-k8s.io -o jsonpath='{.spec.versions[*].name}' tells you which you have. NVIDIA's KAI scheduler (the open-sourced core of Run:ai) models the same idea as a hierarchical Queue with a quota (the deserved floor) and an overQuotaWeight (your proportional claim on idle GPUs above it), plus time-based fair-share so a team that under-used earlier gets favored later.

the endpoint nobody locked

The most common GPU-cluster security hole isn't exotic. It's a vLLM pod with a Service and no authentication, reachable from anywhere on the cluster or, worse, the internet: free inference, prompt exfiltration, model theft. Lock it at three layers. Default-deny ingress so only the gateway can reach the model server:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: default-deny-ingress, namespace: team-a }
spec:
  podSelector: {}
  policyTypes: ["Ingress"]

Then an authenticated, TLS-terminating gateway in front (Gateway API HTTPRoute with auth attached upstream), and a least-privilege ServiceAccount so a compromised inference pod can't read the rest of the cluster's secrets:

apiVersion: v1
kind: ServiceAccount
metadata: { name: vllm-sa, namespace: team-a }
automountServiceAccountToken: false     # the model server never calls the API server

Which connects to the last quiet failure: secrets. Gated models return a 403 at download time when the pod has no valid Hugging Face token, and that surfaces as exactly the CrashLoopBackOff from part 7, except this time it really is the app. Mount the token from a Secret (or an external manager), never bake it into the image, and pin the model by digest so you don't silently load a tampered checkpoint:

env:
- name: HF_TOKEN
  valueFrom:
    secretKeyRef: { name: hf-token, key: token }

Prefer .safetensors over pickle formats (loading a pickle can execute arbitrary code), pin the model revision to a commit, and verify its checksum. The supply chain for a 140GB weight file deserves the same suspicion as any other dependency, which is the whole thesis of the lazy-security series if you want the longer version.

the frontier: confidential computing

One emerging piece, worth knowing exists even if you're not deploying it yet. Confidential computing extends a CPU trusted execution environment to the GPU, so even the cloud operator hosting your node can't read the weights or activations in VRAM. H100 CC-mode is GA: the driver, running inside a confidential VM, encrypts everything crossing the PCIe bus, and CUDA apps run unmodified once trust is established. Blackwell extends it with NVLink encryption for multi-GPU confidential domains. Independent benchmarks put the overhead under 5% for typical LLM inference (it's the CPU-to-GPU I/O encryption that costs, so small-batch workloads pay a bit more). On Kubernetes it lands through Kata Containers and Confidential Containers, and it's still maturing operationally. For regulated or sensitive-IP inference, it's the direction; for most workloads today, it's a section to file away.

who's paying for the idle GPUs

The last shared-cluster problem is money, and it's the biggest one, because cluster GPU utilization commonly sits at 30 to 40%. The gap between GPUs bought and GPUs doing work is enormous, and the fix is making teams see their own idle GPU-hours. Enforce a team label on every GPU pod (a policy engine like Kyverno rejects pods without one), export per-GPU metrics with DCGM, and attribute GPU-hours per team:

validate:
  message: "GPU pods must carry a 'team' label for chargeback."
  pattern:
    metadata:
      labels: { team: "?*" }

With the label enforced, DCGM's per-pod metrics roll up into GPU-hours per team, and tagging inference requests with an X-Team header at the gateway takes it down to token-level attribution. Showback doesn't reclaim a single GPU by itself. It just makes the waste visible to the people who can, which turns out to be most of the battle.

That's eight parts, and it's time to say the thing the whole series was circling. A GPU deployment is a dozen layers under one pod, a small network in one box, a large one between boxes, a set of graphs that tell you the truth, an autoscaler that turns it off, a router that doubles it for free, a set of probes that keep it breathing, and a tenancy model that decides who it hurts when it breaks. The silicon was the easy part. Everything that makes it hard lives in the layers around it, and every one of those layers is a place you can be the person who saw it coming, or the one explaining it in the incident review. That was the job the entire time. The GPUs were never the point.

Your model isn't crashing, your probe is

Harshit Luthra — Thu, 02 Jul 2026 19:15:33 +0000

Originally published at harshit.cloud on 2026-07-23.

The pods were in CrashLoopBackOff and the logs said nothing. vLLM started, printed its usual banner, began loading weights, and then died. Restarted, loaded again, died again. It looked exactly like a broken build, so the first hour went into the model, the image, the CUDA version, everything except the actual culprit, which was a fourteen-line probe config nobody had looked at.

This is part 7. The first six parts built the fleet, wired it, scaled it, watched it, and routed to it. This part is about keeping a model server alive through the three things that routinely kill it: a health probe that fires too early, a node drain that cuts a request in half, and a model-version rollout that drops traffic on the floor. None of it is glamorous. All of it is what stands between you and a 3am page.

your model isn't crashing, your probe is

Here's the tell, and once you've seen it once you never miss it again:

vLLM binds its HTTP port almost immediately, but /health on :8000 only returns 200 once the weights are loaded and the engine is warm. For a 70B on a cold pull that's several minutes. A default liveness probe with a small initialDelaySeconds starts checking during that window, gets a connection refused, decides the container is dead, and the kubelet kills it. It never finishes loading, so it never passes, so it loops forever. The exit code is 137 (128 + SIGKILL) which is the kubelet's fingerprint, not an application crash. That same code turns up for four other reasons outside model serving, and separating them is a two-command job: CrashLoopBackOff: five causes and how to tell them apart.

The fix is a startupProbe. It holds the liveness and readiness probes off entirely until it succeeds, and its failureThreshold × periodSeconds is your total load budget:

startupProbe:
  httpGet: { path: /health, port: 8000 }
  periodSeconds: 10
  failureThreshold: 60      # 60 × 10s = 600s (10 min) to finish loading
livenessProbe:
  httpGet: { path: /health, port: 8000 }
  periodSeconds: 10
  failureThreshold: 3       # after startup, catch a real hang in 30s

The reason this beats just cranking initialDelaySeconds: 600 on the liveness probe is that once the startup probe passes, liveness reverts to its tight interval and still catches a genuine deadlock in thirty seconds. A giant liveness delay would blind you to real hangs for ten minutes after every restart. With the startup probe in place, the pods ride through the load and come up clean:

$ kubectl get pods -l app=vllm-llama3-70b
NAME                               READY   STATUS    RESTARTS   AGE
vllm-llama3-70b-7f4b9c6d8f-9m2tq   1/1     Running   0          8m03s
vllm-llama3-70b-7f4b9c6d8f-c8xvn   1/1     Running   0          8m03s

Key Insight: A model that takes minutes to load needs a startupProbe, not a bigger liveness delay. Without one, the probe that's supposed to detect a dead server is the thing killing a healthy one, and the crash loop looks exactly like an application bug. This is the single most common self-inflicted LLM serving outage.

draining without dropping a stream

The next one bites on every deploy and every node scale-down. When Kubernetes deletes a pod, it sends SIGTERM and removes the pod from the service endpoints at the same instant. vLLM handles SIGTERM correctly (it stops accepting new requests and finishes the in-flight ones), but if the load balancer is still routing to it during that beat, new requests land on a server that's shutting down. And if the grace period is too short, the kubelet SIGKILLs the process mid-generation, dropping a half-finished response the client has to retry from scratch.

Two fields fix it. A preStop hook that sleeps long enough for the endpoint removal to propagate before SIGTERM reaches vLLM, and a terminationGracePeriodSeconds set above your longest in-flight generation:

terminationGracePeriodSeconds: 210   # preStop(15s) + longest decode(~180s) + margin
lifecycle:
  preStop:
    exec:
      command: ["sh", "-c", "sleep 15"]

The sequence is worth internalizing, because the parallelism is the point: the endpoint removal and the preStop sleep happen at the same time, so by the time SIGTERM actually reaches vLLM, the load balancer has already stopped sending it traffic.

Fig. 2 · the drain, in order. The whole job of preStop is to buy time for the load balancer to stop routing before vLLM stops answering.

surviving a node drain

Parts 3 and 5 leaned on Karpenter to consolidate and remove idle GPU nodes. That same consolidation, unguarded, will happily evict every replica of a service at once and take it to zero. The guard is a PodDisruptionBudget, and for a small pool of expensive GPU replicas you want maxUnavailable: 1 so at most one goes down for any voluntary disruption:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: vllm-llama3-70b-pdb
spec:
  maxUnavailable: 1
  selector:
    matchLabels: { app: vllm-llama3-70b }

Karpenter drains through the Eviction API, which respects PDBs, so this is enough to keep the service serving through a consolidation. For a pod that's mid-critical-work and must not be interrupted at all, there's a stronger lever, the karpenter.sh/do-not-disrupt annotation, which excludes its node from consolidation entirely:

metadata:
  annotations:
    karpenter.sh/do-not-disrupt: "true"

Use it deliberately, though. Leave it on permanently and you've told Karpenter it can never reclaim that node, which is how you end up back in part 5's problem of expensive GPUs that never scale down. The PDB is the always-on floor; the annotation is for pods you're actively protecting.

one replica, many pods

When a model is too big for one node (the 405B from part 3, tensor-parallel across eight GPUs and pipeline-parallel across two nodes), a single replica is a group of pods, and a normal Deployment can't express that. LeaderWorkerSet can. It treats a leader plus N-1 workers as one unit: replicas is the number of these groups, size is the pods per group, and RecreateGroupOnPodRestart means if any pod in the group dies the whole group restarts, which is correct, because a tensor-parallel replica missing one member is dead weight:

apiVersion: leaderworkerset.x-k8s.io/v1
kind: LeaderWorkerSet
metadata:
  name: vllm
spec:
  replicas: 2                    # two independent model replicas
  leaderWorkerTemplate:
    size: 2                      # 2 pods each: one leader + one worker
    restartPolicy: RecreateGroupOnPodRestart

The leader starts the Ray head and vLLM's OpenAI server with --tensor-parallel-size × --pipeline-parallel-size equal to the total GPUs across the group; the workers join the Ray cluster via the injected LWS_LEADER_ADDRESS. The catch from part 2 still applies with force: the group has to land on NVLink-connected GPUs or the cross-node collective crawls, so pair LWS with gang scheduling and topology-aware placement (Kueue's TAS, or the KAI scheduler). Rolling updates are first-class through rolloutStrategy with maxUnavailable and maxSurge, so you can roll a multi-node replica without taking the whole service down.

rolling out a new model without an outage

The last way to drop traffic is deploying a new model version badly. KServe makes the careful version cheap: set canaryTrafficPercent on an InferenceService and point storageUri at the new weights, and it splits traffic between the current good revision (which it tracks automatically) and the new one.

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: llama3-chat
spec:
  predictor:
    canaryTrafficPercent: 10       # 10% to the new revision, 90% stays on last-good
    model:
      storageUri: "s3://models/llama3-chat/v2"

You watch the split live, and the metrics from part 4 (TTFT, error rate, and any quality signal) decide whether it graduates:

$ kubectl get isvc llama3-chat
NAME          URL                       READY   PREV   LATEST   LATESTREADYREVISION
llama3-chat   http://llama3-chat...     True    90     10       llama3-chat-predictor-00002

Promotion is deleting the canaryTrafficPercent field and re-applying: all traffic shifts to the new revision and the old one scales to zero. Blue-green is the same mechanism flipped straight to 100; shadow is mirroring live traffic to the candidate without returning its responses, so you can compare outputs with zero user risk. Two things to know before you rely on it. First, this traffic-splitting is a serverless (Knative) mode feature; in raw deployment mode you split with Gateway API route weights instead. Second, version the weights, the serving config, and the prompt template together as one revision, or your canary metrics are comparing two things that differ in ways you didn't track. (And don't reach for ModelMesh for multi-model serving; the project is archived.)

That's the service staying up through loads, drains, and deploys. The last thing between you and a calm on-call isn't the software at all, it's other people: the tenants sharing your cluster, the ones who can reach your endpoint, and a GPU-memory boundary that turns out to be nowhere near where you think it is. That's the final part, and it's the one most likely to end up in an incident review.

The cheapest speedup is your load balancer

Harshit Luthra — Thu, 02 Jul 2026 19:14:53 +0000

Originally published at harshit.cloud on 2026-07-18.

A team I know had a slow chatbot and did the obvious thing: added replicas. TTFT barely moved. They added more. Still slow, and now the bill was worse. Every GPU showed healthy utilization, the queue metrics from part 4 were climbing, and adding capacity wasn't buying the speedup capacity is supposed to buy. Then someone swapped the Kubernetes Service in front of the pods for a prefix-cache-aware router, changed nothing else, and the same eight GPUs got more than twice as fast. The problem was never the GPUs or the count. It was the load balancer, quietly throwing away the most expensive thing the servers had built.

This is part 6. Parts 1 through 5 built the fleet, watched it, and scaled it. This part is about the layer in front of the fleet: how requests get assigned to replicas, and why the default answer (round-robin, the same load balancing you'd use for a stateless web app) is the wrong one for LLM inference. It's the highest ratio of payoff to effort in the whole series, because the win comes from a routing decision, not from hardware you have to buy.

the load balancer that throws away your cache

Here's the thing a normal load balancer doesn't know: LLM replicas are not stateless. Each vLLM replica keeps its own KV cache (from part 2, the running memory of tokens it has already processed), and it also keeps a prefix cache: if two requests share the same opening tokens (a long system prompt, a RAG document, the history of a chat), the second one can reuse the first one's cached computation and skip prefill entirely. Prefill is the compute-bound, expensive half of a request. A prefix cache hit makes it nearly free.

But each replica has its own prefix cache. A round-robin load balancer scatters requests across replicas blind to what each one holds. So a request that shares a 2,000-token prefix with something served thirty seconds ago lands on a different replica that never saw it, re-runs the entire prefill from scratch, and fills its KV cache doing redundant work. Multiply that across a prefix-heavy workload and the fleet spends most of its compute re-prefilling prompts it already processed, on a different pod. The caches sit there full of answers to questions that keep getting routed elsewhere.

the numbers, from one honest benchmark

This isn't a theoretical gain, and the cleanest measurement of it is a benchmark Andy Golubev ran on EKS in June 2026. The setup was deliberately boring: Qwen2.5-7B on vLLM, eight g5.xlarge nodes with one A10G each, one decode replica per node. Two runs, identical in every way except the front door. First run: a plain Kubernetes Service doing round-robin. Second run: an llm-d prefix-cache-aware router. Same model, same eight GPUs, same eight replicas. The workload was vllm bench serve replaying 9,000 prompts that shared a pool of 150 long (2,048-token) prefixes, which is exactly the RAG-and-chat shape real traffic takes.

The gap was not subtle.

Fig. 2 · the Golubev EKS numbers. The one that matters most is the last one: round-robin ran the cache pinned at 99% and thrashing; the smart router left it headroom, because it stopped generating redundant work.

Read the cache-hit line again: 11% to 93%. Round-robin was reusing almost nothing; the aware router reused almost everything. Mean TTFT dropped from a genuinely broken 19 seconds to under a second. And the fleet stopped queueing (waiting requests went from ~180 to zero) not because it got more capacity, but because it stopped wasting the capacity it had. Google reports the same shape from GKE's managed version: TTFT improvements up to 96% at peak load on prefix-heavy workloads. The lesson is uncomfortable and freeing at once. Before you buy a ninth GPU, check whether your router is making the eight you have re-do each other's work.

routing to the replica that already knows

The mechanism is simpler than it sounds. vLLM emits events about what its prefix cache holds. A cache-aware router consumes those events and keeps a live picture of which replica has which prefixes cached. When a request arrives, instead of picking the next replica in rotation, the router picks the one most likely to already hold the request's prefix, so the cache hit actually happens.

That decision point has a name in the Kubernetes world: the Endpoint Picker, or EPP. It doesn't just look at prefix locality. A good EPP scores every candidate replica on several signals at once and sends the request to the best total score:

prefix-cache locality: does this replica already hold the request's opening tokens? Longer match, higher score.
load: what's this replica's KV-cache utilization and queue depth right now? A replica that's already saturated scores lower even if it has the prefix, so you don't stampede one hot pod.
LoRA affinity: if you serve multiple fine-tuned adapters, is the right adapter already loaded here? Loading one is not free, so prefer the replica that has it.

The router balances those against each other, and it can queue or shed when everything is overloaded. This is the difference between "which pod is next" and "which pod will serve this request fastest given what it already has warm." Same request, same pods, a decision made with information the round-robin balancer never had.

the standard nobody had two years ago

The reason this is worth a whole chapter now, and wasn't in 2024, is that it stopped being a bespoke hack and became a Kubernetes standard. The Gateway API Inference Extension is a SIG-Networking project that adds LLM-aware routing on top of the ordinary Gateway API. It introduces an InferencePool, which is a group of pods that share the same accelerator, base model, and model server (the LLM-shaped version of a Service), and it wires an Endpoint Picker into the request path through Envoy's external-processing (ext-proc) protocol, so the proxy calls out to the EPP for a decision on every request.

Fig. 3 · the request path. The gateway is a normal Envoy proxy; the intelligence lives in the Endpoint Picker it consults per request, over the same ext-proc hook Envoy already uses for auth and rate limiting.

The project has reached GA, with InferencePool graduating to a stable v1 API. (One honest caveat: the exact GA milestone landed over early-to-mid 2026, so pin the version you're deploying rather than trusting a blog's date, this one included.) Google productized it as GKE Inference Gateway, which is GA and is literally powered by the llm-d router underneath. So the "advanced" thing here is also increasingly the default, k8s-native thing, which is exactly the production framing this series cares about.

the whole stack, named

"llm-d" gets thrown around as if it were a server. It isn't; it's an assembly, and it's clearer to name the pieces:

vLLM is the model server. It owns the per-replica KV and prefix cache and emits the cache events the router needs.
KServe is the serving control plane, exposing an LLMInferenceService custom resource so you describe the model and its serving config as a normal Kubernetes object.
The inference gateway is Envoy plus the Gateway API Inference Extension: the data plane plus the LLM-aware routing above.
The router itself is that L7 proxy plus the Endpoint Picker, making the per-request decision from cache, load, and LoRA signals.
Disaggregated prefill/decode from part 3 is an optional add-on here, with the KV cache handed between pools over a connector.

If you're not ready to adopt the full gateway, the vLLM project's own production-stack helm chart is a lighter on-ramp: it deploys a router service in front of your vLLM pods that already does model-aware and prefix-aware routing, plus KV-cache offload through LMCache. Same idea, smaller commitment. Either way, the thing you're installing is a router that knows what a KV cache is.

watching the router

Part 4 said the metrics that matter live in the serving engine. Add one more surface: the router. The signals that tell you whether the routing layer is earning its keep are the ones that moved in that benchmark. Prefix cache hit rate is the headline. A round-robin fleet sits low (that 11%); a well-routed prefix-heavy fleet should sit high (the 93%). If you turned on cache-aware routing and the hit rate didn't climb, either your workload doesn't actually share prefixes or the router isn't seeing the cache events.

Key Insight: Prefix cache hit rate is the one dashboard number that proves the router is working. Watch it alongside per-replica KV utilization and waiting-request count. If hit rate is high and evenly spread, routing is doing its job. If one replica is pinned while others idle, your scorer is over-weighting cache locality and stampeding a hot pod. If hit rate is flat near zero, your traffic isn't prefix-heavy and this whole chapter buys you little.

That last line is the honest boundary. Cache-aware routing is close to free money for workloads with real prefix reuse: RAG over a shared corpus, long system prompts, multi-turn chat, agents replaying context. For traffic where every prompt is unique and short, there's no cache to reuse and the fancy router mostly just adds a hop. The KV-connector interfaces underneath are also still moving, so expect some churn if you build on the bleeding edge. Know which workload you have before you reach for this, the same way you'd check the topology in part 2 before promising a throughput number.

The router is the front door. What sits behind it, the probes that decide when a replica is ready, the way you roll a new model version out without dropping a request, the tenants all fighting over the same pool of GPUs, is where a production inference platform actually gets hard. That's where the series goes next.

Scaling GPU inference to zero and back

Harshit Luthra — Thu, 02 Jul 2026 19:14:48 +0000

Originally published at harshit.cloud on 2026-07-16.

The finance dashboard is what started it. A staging cluster running two H100 nodes around the clock, mostly to serve a demo that got maybe forty requests a day, all of them during business hours. The nodes sat idle from 7pm to 9am and every weekend, billing the whole time. Somebody added scale-to-zero over a Friday. Monday morning the first person to open the demo waited six minutes for a response, assumed it was broken, and filed a bug. The bill went down and the product got worse, which is scale-to-zero working exactly as designed and nobody being happy about it.

This is the last part of the series. Parts 1 through 4 covered building GPU infrastructure and seeing what it's doing. This part is about the thing that actually shows up on the invoice: a GPU you're paying for while it does nothing. Scaling GPU inference elastically, all the way to zero when there's no traffic, is the biggest lever on the bill. It's also genuinely hard, because unlike a stateless web pod that starts in a second, a GPU replica has to drag a hundred gigabytes of model weights onto the card before it can serve a single token. The whole post is about that gap.

the cold-start tax

Scaling a web service to zero is free because starting a new pod is nearly instant. Scaling an LLM to zero is expensive because starting a replica is not. Add up what has to happen before the first token, on a cold node:

Node provisioning. The cluster asks the cloud for a GPU node, waits for it to boot and join. One to five minutes, and that assumes the GPU is even available to hand you (more on that at the end).
Image pull. A CUDA plus vLLM container image is commonly 5 to 15 GB. On a fresh node with a cold cache, pulling and unpacking it is minutes, not seconds.
Model load. Llama-3-70B in BF16 is about 140 GB of weights, usually sitting in object storage. Reading that over the network and moving it onto the GPU is the big one, and done naively it's several minutes.
Warmup. CUDA graph capture, torch.compile, and a few dummy forward passes to populate caches. Tens of seconds before the first real request is fast.

Stack those up and a naive 70B cold start runs six to nine minutes, more if those 140 GB of weights come cold over a slow path. That's the tax. Everything else in this post is a way to stop paying it, so that scale-to-zero saves the money without the six-minute Monday.

One cruel interaction hides in here: a naive Kubernetes health probe treats that multi-minute load as a failure. A liveness probe with default timing restarts the pod mid-load, and now you have a crash loop that looks like a vLLM bug but is really a probe firing too early. Set the liveness initialDelaySeconds above your worst-case load time, keep it longer than the readiness delay, and add a preStop sleep so in-flight requests drain before the pod goes down. It's the single most common self-inflicted serving outage, and it costs nothing to avoid once you've seen it once.

scale on the queue, not the GPU

Before scaling to zero, get scaling to anything right, and the first mistake is scaling on GPU utilization. It's the obvious metric and it's the wrong one, for the reason part 4 laid out: a GPU can sit at 95% while the queue is empty, or at 40% while requests pile up. Scaling on GPU-util adds replicas late and removes them at the wrong time.

Scale on the serving signal instead. The metric that actually tracks unmet demand is the queue: vllm:num_requests_waiting, or KV-cache utilization as a leading indicator. You'll still find plenty of setups scaling on GPU-util or cache percentage, and those aren't wrong so much as indirect; the point is to scale on demand rather than on how busy the chip happens to look, with tail latency as the guardrail. In practice that means KEDA (the Kubernetes event-driven autoscaler) with a Prometheus trigger reading that metric, or an HPA wired to the same value through the Prometheus adapter. For traffic that's genuinely request-driven and spiky, KEDA's HTTP add-on can scale on in-flight request count directly. The shape is the same: pick the metric that means "users are waiting," set the threshold at the per-replica capacity you measured with the load test in part 4, and let it add replicas before the queue becomes TTFT.

actually reaching zero

Scaling to zero is a special case of scaling, with one extra problem: when you're at zero replicas, there's nothing running to receive the request that's supposed to wake you up. Something has to catch that first request and hold it while a replica spins up.

KEDA does this with an activation threshold: below it, the deployment sits at zero; the first event scales it to one. Knative Serving builds the pattern in more deliberately, with an activator component that buffers incoming requests while a cold replica starts, then releases them once it's ready, so the request is slow but not dropped. KServe (which runs on Knative) exposes this as a simple minReplicas: 0 on an InferenceService, and it's the most common way teams get GPU model servers to zero on Kubernetes.

Fig. 2 · the lifecycle, and the unlucky first request. Everything after this figure is about shrinking the cold-start box so that request waits seconds, not minutes.

The honest catch is that reaching zero and the cold-start tax are the same coin. Zero replicas is where the savings are, and it's also where every request pays the full six minutes. Everything below is about making that first request cost seconds instead, because a scale-to-zero setup with a six-minute cold start is a cost win and a product loss, and you rarely get to keep both.

There's also a breakeven worth running before you build any of this. Scale-to-zero wins when utilization is low and spiky; it loses somewhere around half-time, where a dedicated node is both cheaper than paying serverless rates by the second and free of cold starts entirely. If a GPU is busy more than half the day, don't scale it to zero. Right-size it and leave it on.

killing the image pull

The container image is the first fixable minute. Two families of fix.

The first is to stop pulling the whole thing before you start. SOCI (Seekable OCI), lazy-loading via a containerd snapshotter, lets a container start running against an index of the image and fetch the actual bytes on demand, so the model server boots while the layers it doesn't need yet are still downloading. estargz (the containerd stargz snapshotter) and Nydus do the same lazy-pull trick with different formats; GKE's Image Streaming is the managed version, and it cut a 5.4 GB Triton image's start from 191 seconds to 30. There's also a variant for the case where you will read the whole image anyway (AI images touch most of their bytes immediately): SOCI's parallel-pull mode just parallelizes the download and unpack instead of lazy-loading, and AWS measured roughly 60% off a 10 GB image.

The second is to not pull over the network at all. Pre-bake the image into the node's disk image so it's local before the pod schedules. Run an in-cluster pull-through cache or a peer-to-peer image mirror (Spegel and friends) so the second node to need an image gets it from a neighbor instead of the registry. Pre-pull hot images with a DaemonSet so they're warm before traffic arrives. None of these is clever; all of them beat pulling 15 GB cold from a registry when a hundred replicas try it at once.

killing the model load

The bigger minute is the model weights, and this is where the newer tooling has moved fast. Loading 140 GB of safetensors off a disk or object store the default way is serial and slow. The fixes stream instead.

NVIDIA's Run:ai Model Streamer reads weights from object storage in many parallel streams straight onto the GPU, overlapping download with load, and vLLM supports it directly (--load-format runai_streamer). NVIDIA's own benchmark took an 8B model's S3 load from 28 seconds at four streams to under five at thirty-two. CoreWeave's tensorizer (--load-format tensorizer) serializes the model into a format that streams from S3 or local disk with near-zero deserialization overhead. Both turn a multi-minute load into tens of seconds. Underneath, safetensors already supports memory-mapped zero-copy loading, which helps when the file is local.

And local is the other half. Cache the weights on the node's NVMe (an instance-store disk) so a restart reads from local flash instead of re-downloading. Or mount a shared, fast filesystem so every replica reads the same warm copy, but mind the access mode: a ReadWriteOnce PVC serves one replica and then silently blocks the second pod from mounting, so anything that scales past a single replica needs ReadWriteMany (EFS, FSx for Lustre, NFS, CephFS). That RWO-to-RWX switch is a classic thing to discover the first time an autoscale event never becomes a second pod. Some teams ship the model as its own OCI artifact and let the image machinery above handle it (KServe's modelcar pattern). The principle is the same one from part 2: the bandwidth between the weights and the GPU is the bottleneck, so shorten that path.

The frontier technique skips loading altogether. GPU memory snapshots (Modal and Cerebrium both ship this, built on the CUDA checkpoint/restore API in recent driver branches) checkpoint a fully warmed replica, weights on the card and CUDA graphs already captured, then restore that image straight onto a GPU. Because it bypasses weight load, torch.compile, and graph capture in one move, it's the only approach that also kills the warmup tax. Modal reports a vLLM Qwen model dropping from 45 seconds to 5, and Cerebrium measured cold starts down 71 to 88 percent. The catch is portability: a snapshot is pinned to a specific GPU model and driver branch, so it's a per-SKU artifact, not a universal one.

Key Insight: Scale-to-zero is a tradeoff, not a free win. Every second of cold start is latency the first user eats; every minute of warm idle is money you burn. You buy down the cold start with lazy image pulls, model streaming, and a warm node pool, then set a minimum replica floor high enough that your p99 cold start stays inside the SLO. "Zero when truly idle, one when it might not be" beats a dogmatic zero.

Fig. 3 · the fix menu, one column per stage. Snapshots are the only trick in the third column, which is why they're the frontier: everything else leaves the warmup tax standing.

turning off idle nodes

Scaling pods to zero doesn't save anything if the expensive GPU node they were on keeps running. The node has to go too. This is the autoscaler's job below the pod level: Karpenter consolidates workloads and removes nodes that are empty or underutilized (consolidateAfter, disruption budgets so it doesn't yank capacity mid-request), and the older cluster-autoscaler scales node groups down on the same principle. A GPU node that's been empty for a few minutes is thirty to a hundred-plus dollars a day; letting it linger is the single most common way GPU bills quietly balloon.

For predictable traffic, the laziest win is scheduled scaling. If the demo only serves business hours, a cron trigger that scales the floor to zero at 7pm and back to one at 9am captures most of the savings with none of the cold-start risk during the day. And to hide cold starts when you do scale up, keep a warm node in reserve with low-priority placeholder pods (over-provisioning): real work evicts the placeholders instantly and lands on a node that already has the image, so the pod cold start doesn't also pay the node cold start.

spot, and the capacity trap

Two closing realities that scale-to-zero runs into. Spot instances make idle capacity cheap, and for inference they can work (unlike the gang-scheduled training from part 3, a single inference replica dying is survivable). But a spot GPU can be reclaimed with about two minutes' notice, so you need a node-termination handler that drains in-flight requests and a plan for where the replacement comes from.

Which is the trap: scaling up assumes there's a GPU to scale up onto. In 2026, popular GPUs are not always available on demand, and a scale-to-zero service that can't reacquire an H100 at 9am Monday is worse than one that never scaled down. The mitigations are the capacity blocks and reservations from part 3, an on-demand fallback when spot is dry, and a warm floor for anything with a real SLO. Scale-to-zero is a cost strategy, not a capacity strategy, and confusing the two is how you save money right up until the morning you can't get your GPUs back.

That's the cost side handled. The fleet scales with demand and turns itself off when it's idle, and the first user back doesn't wait five minutes for the privilege. But there's one lever left, and it's the strangest one, because it costs nothing to pull. Everything so far has quietly assumed that a request, once it arrives, lands on some replica and gets served. Part 6 is about that word "some." It turns out that which replica you pick, out of a pool that all look identical from the outside, can make the exact same hardware more than twice as fast. The load balancer, of all the unglamorous things, is where the last big win hides.

What a green GPU dashboard hides

Harshit Luthra — Thu, 02 Jul 2026 19:14:09 +0000

Originally published at harshit.cloud on 2026-07-11.

The GPU dashboard was a wall of green. Every card pinned at 90-something percent utilization, power near TDP, temperatures fine, no XID errors. By every signal from part 1 of this series, the box was healthy and working hard. Meanwhile the on-call channel had three messages from product asking why the chatbot took eight seconds to say its first word. The hardware was busy. The users were furious. Both were true.

That is the trap of monitoring GPU inference with GPU metrics. A card at 92% utilization tells you a kernel is running. It tells you nothing about whether requests are piling up in a queue, whether the KV cache is full and requests are being evicted, or whether the first token is landing in 200 milliseconds or eight seconds. This is part 4 of the series. Parts 1 through 3 built the thing: the stack under a pod, the wires in a box, the network between boxes. This part is about seeing what it's doing once real traffic hits it, and the short version is that the numbers users feel live in the serving engine, not on the GPU.

the four numbers users actually feel

Before any dashboard, get the vocabulary straight, because LLM latency is not one number. A request has a shape, and four measurements describe it.

TTFT, time to first token, is how long the user stares at a blinking cursor before anything appears. It's dominated by prefill (processing the whole prompt) plus however long the request sat in a queue before the server picked it up. This is the number product complains about.

ITL, inter-token latency, is the gap between consecutive tokens once generation starts. (Some tools report TPOT, time per output token, the averaged version of the same thing; benchmarks like GuideLLM show both side by side, so don't treat them as interchangeable in a report.) It's what makes text feel like it's streaming smoothly or stuttering out. Decode is memory-bandwidth-bound, so ITL degrades as you pack more concurrent requests onto a GPU.

End-to-end latency is the whole request, first byte to last. For a long generation it's mostly output_tokens × ITL, which means it's as much about how much the model says as how fast it says it.

Throughput is output tokens per second across all requests, and it moves in the opposite direction from latency. Batch more requests together and throughput climbs while per-request latency gets worse. There's no single "fast" setting. There's a frontier, and where you sit on it is a product decision (a chatbot wants low TTFT, a batch summarization job wants raw throughput) rather than a tuning default.

The number that captures both at once is goodput: the requests per second you can serve while still meeting your latency SLO. A server can post gorgeous raw throughput while quietly blowing p99 TTFT for half its users, so goodput (throughput, filtered by "did it meet the SLO") is the only throughput figure worth quoting.

what vLLM actually tells you

vLLM exposes a Prometheus /metrics endpoint, and once you've read it a few times the health of the server is obvious at a glance. The metrics that matter split into two groups: what the queue is doing, and what the KV cache is doing.

The queue first. vllm:num_requests_running is how many requests are being served right now; vllm:num_requests_waiting is how many are stuck in line because the server can't fit them yet. A healthy server has a running count near its batch capacity and a waiting count near zero. When num_requests_waiting starts climbing and staying up, that eight-second TTFT has arrived, and no GPU metric will show it. vllm:request_queue_time_seconds measures the wait directly.

Then the KV cache (the model's running memory of the tokens it has already processed, which grows with every active request and every token generated). vllm:kv_cache_usage_perc is the fraction of KV cache in use. This is the one to stare at, because when it approaches 100% vLLM has to start preempting: evicting a half-finished request to free memory, then recomputing it from scratch later. vllm:num_preemptions_total counts that happening. A rising preemption rate means the server is thrashing, doing the same work twice, and every latency number is about to get worse.

Fig. 2 · the same server, healthy and drowning. GPU utilization is high in both; the queue, the cache, and the preemption counter are what separate a busy server from a failing one.

The rest fill in the picture. vllm:time_to_first_token_seconds and vllm:inter_token_latency_seconds are histograms, so you alert on the p95 or p99, not the average (the average hides the user who waited twelve seconds). vllm:prompt_tokens_total and vllm:generation_tokens_total give you real throughput, computed with rate() (vLLM removed its old pre-averaged throughput gauges, so you do the division yourself). And vllm:prefix_cache_hits_total over vllm:prefix_cache_queries_total tells you how much prompt reuse you're getting, which matters enormously for the RAG and multi-turn workloads from part 2. It's also the number a smart router watches to decide which replica to send a request to, which is the whole subject of part 6.

One warning that will save you an afternoon: these names drift between releases. The V1 engine renamed the KV-cache gauge (it used to be gpu_cache_usage_perc) and swapped the per-output-token metric to inter_token_latency_seconds. Diff the live /metrics output of your exact build before you copy anyone's PromQL, this post included.

Key Insight: GPU utilization tells you the chip is busy; it never tells you users are waiting. The metric that predicts an angry inbox is vllm:num_requests_waiting climbing, usually because vllm:kv_cache_usage_perc hit its ceiling and the server started evicting half-finished requests. Watch the queue and the cache, not GPU-util.

the same story in SGLang and Triton

The engine changes, the questions don't. SGLang exposes its own Prometheus metrics behind --enable-metrics: sglang:num_running_reqs and sglang:num_queue_reqs are the running-and-waiting pair, sglang:token_usage is the KV-cache fraction, and sglang:cache_hit_rate reports how often its RadixAttention prefix cache paid off. (One gotcha of exactly the kind above: SGLang flipped the metric prefix from sglang: to sglang_ in v0.5.4, and the bundled Grafana dashboard hasn't caught up, so a fresh install can read "No Data" until you fix the prefix.) If you built on SGLang for its prefix-sharing (the reason to pick it in part 2), that last one is how you confirm the bet is paying.

Triton with the TensorRT-LLM backend reports through Triton's metrics endpoint instead: request and queue durations, inflight-batcher stats, per-model success and failure counts. Different names, same three questions every time: how long are requests waiting, is the cache saturated, is the tail latency inside the SLO.

The lesson worth internalizing is that these are all the same dashboard with different labels. Queue depth, cache utilization, tail TTFT, tail ITL. Learn to read one engine and you can read all of them.

compute-bound, memory-bound, or queue-bound

The reason to keep the GPU metrics from part 1 next to the serving metrics is that together they diagnose why the server is slow, which is the only thing that tells you what to do about it. Three shapes cover most incidents.

If TTFT is high and num_requests_waiting is high but gpu_cache_usage_perc is low, you're queue-bound: requests are backing up faster than you can start them, and the fix is more replicas (which is part 5). If ITL is degrading and DCGM_FI_PROF_DRAM_ACTIVE is pinned while tensor activity isn't, you're memory-bound on decode, and the fix is a smaller batch, quantization, or better KV-cache management. If tensor cores are saturated during prefill and DRAM isn't, you're compute-bound, which for inference usually means very long prompts and points at chunked prefill or a prompt-length limit.

Fig. 3 · the serving metric tells you something is wrong; the GPU metric next to it tells you which kind of wrong. You need both panels on the same screen.

Neither set of metrics is enough alone. GPU metrics without serving metrics miss the queue entirely. Serving metrics without GPU metrics can't tell a memory-bound stall from a compute-bound one. The dashboard that works has both, side by side, on one screen.

what to alert on

Most teams over-alert on the GPU and under-alert on the experience. The page that matters fires on the user's SLO, not the hardware's vitals. A working starter set:

TTFT p99 over budget for N minutes. This is the customer-facing SLO. Everything else is a leading indicator of this.
num_requests_waiting sustained above zero. A brief spike is fine; a standing queue means you're under-provisioned and the next thing to break is TTFT.
Preemption rate climbing. num_preemptions_total moving means the KV cache is saturated and the server is recomputing evicted work. It's the early warning before latency falls off a cliff.
Error rate. Request failures and, specifically, CUDA out-of-memory events, which on an inference server usually mean a batch or context-length setting is too aggressive.
The part-1 hardware alerts still stand. XID errors, thermal throttle, ECC. A dying GPU shows up as latency variance long before it shows up as an error, so keep those wired.

Tail latency is the whole game here. Alerting on average TTFT is how you find out about an outage from the customer instead of the pager, because the average stays calm while your p99 is on fire.

tracing a single slow request

Aggregate metrics tell you the fleet is unhealthy. They don't tell you why this request took nine seconds. For that you want per-request tracing, and the ecosystem has standardized on OpenTelemetry's GenAI semantic conventions: spans carry gen_ai.* attributes (the model, input and output token counts, the request parameters) so a single request's journey through the gateway, the queue, prefill, and decode is one connected trace. When a specific user reports a slow response, a trace tells you whether it sat in a queue, hit a cache miss, or just asked for a 4,000-token essay. The metrics say the kitchen is slow; the trace shows you which order got lost. Which of these should wake someone is a separate decision, and worth making on the error budget rather than a utilisation threshold: alert on the error budget, not the CPU graph.

proving it before prod

You don't want to discover your latency frontier during a launch. Load-test the serving endpoint before it sees real traffic, with a tool that speaks LLM rather than plain HTTP. vLLM ships vllm bench serve (the old benchmark_serving.py), which replays a request distribution, reports TTFT, ITL, and throughput percentiles, and computes goodput directly if you hand it SLO thresholds. GuideLLM (now a Red Hat project) does the same with a sweep mode that finds your safe operating range on its own; NVIDIA's genai-perf covers the Triton side, though NVIDIA is steering new work to its successor AIPerf; and the Kubernetes serving working group's inference-perf standardizes the numbers across engines so you can compare vLLM to SGLang honestly. Whatever you pick, the output you care about is the same: the curve of tail latency against offered load, and the load at which p99 TTFT crosses your SLO. That crossover is the goodput ceiling, the real per-replica capacity, and it's the input to everything in part 5.

Because that's the thing this post sets up. Once you can see the queue building and the cache saturating, the obvious next question is: why am I staring at these graphs manually at 2am instead of having the queue depth add a replica by itself, and drop it again when the traffic goes home. That's scaling, and scaling GPUs that cost six dollars an hour is its own kind of problem.

Scaling GPUs past one box

Harshit Luthra — Thu, 02 Jul 2026 19:14:04 +0000

Originally published at harshit.cloud on 2026-07-09.

When Meta trained Llama 3 405B on 16,384 H100s, the cluster hit 419 unexpected interruptions over 54 days. That's one failure every three hours or so, for 54 days straight, and about 78% of them were hardware. GPUs and their HBM3 memory accounted for roughly half. They also watched the datacenter's power draw swing by tens of megawatts as thousands of GPUs idled and resumed in sync, which is a sentence that should make any infrastructure engineer sit up.

This is the part of the series where the GPU stops being the interesting component. Part 1 was the stack under one pod, part 2 was one box and the wires inside it. Once you cross the node boundary, the network becomes the machine, failures become continuous rather than exceptional, and the scheduler decides whether your very expensive cluster does useful work or deadlocks against itself. Everything here is about the stuff between the boxes.

the network is the machine

Synchronous training does an all-reduce of the entire gradient (every parameter, billions of them) every single step. That collective is a hard barrier: the slowest link gates every GPU in the job. Add nodes and your compute scales, but the all-reduce volume grows and so does the chance that one link is slow or dead. This is why, past one node, communication rather than FLOPs sets your scaling efficiency, and why the network gear costs as much attention as the GPUs.

The fabric itself is a two-horse race. InfiniBand is the incumbent for dedicated training superclusters: the generation ladder runs EDR 100Gb, HDR 200Gb, NDR 400Gb (Quantum-2 switches, ConnectX-7 NICs), and now XDR 800Gb (Quantum-X800, ConnectX-8). Most DGX SuperPODs and the big named clusters run it. RoCE v2, RDMA over Ethernet, is winning share on cost and on letting existing Ethernet teams reuse what they know. The catch with RoCE is that it needs a carefully tuned lossless fabric (Priority Flow Control plus ECN marking) or you get congestion storms, and getting that right at scale is its own discipline.

Meta is the proof that Ethernet can do it. They built two 24,576-GPU H100 clusters, one on RoCE and one on Quantum-2 InfiniBand, and trained Llama 3 405B on the RoCE one with no network bottleneck, after co-designing the topology, the PFC/ECN thresholds, and an all-reduce-aware load balancer. That's the honest framing: RoCE works at scale, but Meta spent real engineering to make it work. Two more pieces earn their keep on either fabric. GPUDirect RDMA lets the NIC DMA straight into GPU memory, skipping a bounce through host RAM, and without it every hop stages through system memory. SHARP does the reduction inside the switch ASIC, so gradients get summed in the network instead of shuttled between every node, which on the newest Blackwell fabrics is a large multiplier on effective all-reduce bandwidth.

NCCL across the wire

The same NCCL from part 2 handles inter-node collectives, and the failure mode here is specific and common: NCCL silently falls back to TCP sockets when it can't find or use the RDMA path, and the job "works" while running an order of magnitude too slow. The env vars that prevent that are worth pinning in your launcher.

NCCL_IB_HCA names which RDMA NICs to use, and getting it wrong means NCCL picks one NIC and loses your rail parallelism. NCCL_SOCKET_IFNAME has to point at the real data-plane interface, not eth0 or lo, a classic container and Kubernetes trap. NCCL_CROSS_NIC=0 on a rail-optimized fabric keeps a ring on the same rail instead of hopping across them. NCCL_IB_GID_INDEX is the RoCE gotcha: the wrong GID index gives you no traffic or a silent slow path. On a fresh cluster the first all-reduce is routinely two to ten times slower than optimal until the env vars, the GID index, the PFC/ECN config, and the topology file are all correct. The bring-up ritual is always nccl-tests: run all_reduce_perf, measure the achieved bus bandwidth, compare it against what 400 or 800 Gb should give you, and don't trust the cluster until the number is close.

who schedules the gang

Here's the failure that surprises people coming from web infrastructure. Vanilla Kubernetes schedules pods independently, one at a time, with no concept of a job that needs all its pods at once. Give it a 4-node training job and it will happily place 3 pods and leave the 4th Pending forever, holding three nodes of GPUs idle. Run two such jobs and they can each grab most of what the other needs and starve each other indefinitely. Distributed training is all-or-nothing, and a scheduler that doesn't know that will deadlock your cluster. The first time it happens you assume you're out of GPUs. You're not. They're all sitting idle, reserved by pods that will never get their partners.

Fig. 2 · gang scheduling in one picture. The left side is how a cluster quietly wedges itself; the right side is the fix, and the reason every GPU scheduler below exists.

The fix is gang scheduling: admit all N pods together or none, so partial allocations can't happen. The tools that provide it each have an honest drawback:

Slurm is the HPC default and has gang scheduling and topology awareness built in, plus new block scheduling for aligning jobs to NVL72 racks. Its weakness is that containers and multitenancy are bolted on (Pyxis plus Enroot), and it's a poor fit for long-running inference services. That gap is why Slurm-on-Kubernetes projects like SchedMD's Slinky and CoreWeave's SUNK exist.
Kubernetes vanilla has no gang scheduling and no topology awareness by default. You add a batch scheduler on top; you don't run training on the default scheduler.
Volcano is the de facto CNCF choice: gang scheduling via PodGroups, queues, fair-share. It runs as a second scheduler that bypasses the default, which complicates coexistence with normally-scheduled workloads, and gang scheduling itself costs maybe 10–15% utilization because resources sit idle waiting for the full gang.
Kueue is the Kubernetes-native answer for queueing and quota, and it cooperates with the default scheduler instead of replacing it. The tradeoff is that it does admission and quota, not fine-grained placement, so you still need scheduler plugins underneath for the actual gang and topology binding.
Run:ai is the commercial option, now NVIDIA-owned, with fractional GPU and pooling. NVIDIA open-sourced the core as KAI Scheduler (Apache 2.0, now CNCF Sandbox), so a free path exists, but the full enterprise feature set stays paid and KAI is young as a standalone project.
YuniKorn brings strong hierarchical-queue multitenancy from the Spark world, at the cost of being another full scheduler replacement with a smaller AI-specific ecosystem than Volcano.

The cross-cutting truth is that gang scheduling trades utilization for progress. Holding GPUs idle while you wait for the full gang is the price of not deadlocking, and it's a price worth paying.

splitting the model

When a model outgrows one GPU, there are three axes to split it on, and real training combines them. Data parallelism replicates the whole model on each GPU and all-reduces gradients; it's the simplest and only works when the model plus its optimizer states fit on one card. Tensor parallelism splits individual matrix multiplies across GPUs and is communication-heavy, so you keep it inside a node on NVLink (TP=8 is the common ceiling). Pipeline parallelism cuts the layers into stages across nodes and passes activations point-to-point, which is cheap enough to cross the network.

Fig. 3 · the standard frontier recipe: tensor-parallel inside the NVLink domain, pipeline-parallel across nodes, data-parallel on top. Match each split to the bandwidth it can afford.

For the common case of "my model doesn't fit but I want to stay in native PyTorch," FSDP2 shards the parameters, gradients, and optimizer states across GPUs and reconstructs each layer on the fly via all-gather, prefetching the next shard to overlap communication with compute. DeepSpeed's ZeRO does the same idea in stages: stage 1 shards optimizer states, stage 2 adds gradients, stage 3 adds parameters and is functionally equivalent to FSDP. For frontier scale and maximum MFU you reach for Megatron-Core and combine tensor, pipeline, and sequence parallelism into the 3D (now 4D, with expert parallelism for MoE) recipe: TP=8 inside the node, pipeline across nodes, data parallelism on top.

Checkpointing is the reliability workhorse and used to be the tax that made frequent saves unaffordable. Async distributed checkpointing fixed that by writing state in a background thread that overlaps the next iterations; TorchTitan reports 5–15x lower checkpoint overhead than synchronous saves. That matters directly because of the failure rate: at one interruption every three hours, you want to checkpoint on the order of tens of minutes, and torchrun's elastic mode restarts the job from the last snapshot when a node dies. The newer torchft goes further, recovering a failed replica from a healthy peer without restarting the whole job.

when the model won't fit one node

Serving crosses the node boundary for the same reason training does: the model, or its KV cache, exceeds one node's total HBM. DeepSeek-V3 at 671B, Llama 405B, the big mixture-of-experts (MoE) models. You split them with tensor and pipeline parallelism across nodes, and for MoE you add wide expert parallelism, spreading experts across many nodes so each GPU holds few experts but sees a large batch per expert.

The pattern that's become standard is disaggregated prefill and decode. Prefill (processing the prompt) is compute-bound; decode (generating tokens one at a time) is memory-bandwidth-bound. Running them in one pool means prefill work stalls decode latency. Splitting them into separate worker pools lets you scale each for its own bottleneck and transfer the KV cache between them over RDMA. It isn't a free win, though. Moving the KV cache between pools costs bandwidth, so disaggregation pays off when prefill interference is genuinely the bottleneck (long prompts, high concurrency) and can be net-negative when it isn't. DeepSeek runs it because at their scale it clearly is; a chatbot with short prompts might not need it at all.

Fig. 4 · prefill and decode want different hardware ratios, so modern stacks run them as separate pools and ship the KV cache between them. DeepSeek-V3 runs a 32-GPU prefill unit in front of a 320-GPU decode pool.

DeepSeek's own deployment runs a 32-GPU prefill unit (4 nodes, expert-parallel across 32) feeding a much larger decode pool, and an LMSYS reproduction on 96 H100s hit 52,000 input and 22,000 output tokens per second per node. On Kubernetes the primitive that expresses "this is one model replica made of many pods" is LeaderWorkerSet: one leader, N workers, scheduled and scaled as a unit, which is exactly what gang scheduling and topology-aware placement need to bite on. NVIDIA Dynamo and llm-d sit on top: Dynamo for distributed serving with a KV-cache-aware router, and llm-d for KV-cache-aware routing on the Gateway API Inference Extension. That routing layer turns out to be one of the biggest free wins in the whole stack, which is why it gets its own part 6.

everything fails at scale

Reliability stops being a checkbox and becomes the main event. The Llama 3 numbers from the top of this post are the reference point, and ByteDance's MegaScale run on 12,288 GPUs tells the same story: 55.2% MFU and more than a hundred failure-recovery events over a few weeks. The failure you can't see coming is silent data corruption, where a GPU computes a wrong number without erroring. It doesn't crash or log anything. It just hands back the wrong answer, and your loss curve grows a mysterious kink a few hours later. Meta caught six such events in 54 days; Google reports SDC-related disruptions roughly every one to two weeks. A single corrupted gradient contaminates the global update across every worker, and it's now a first-class reliability topic with its own whitepapers.

Key Insight: At cluster scale, hardware failure is the normal state, not an exception you engineer away. A 16k-GPU run loses a GPU every few hours. You can't stop that, so the whole game is checkpointing often enough and keeping enough hot spares that a dead node costs you minutes instead of the whole run.

The straggler is the SDC's cousin: a GPU or link that's degraded but not dead, quietly throttling a synchronous job because collectives move at the speed of the slowest member. Detecting it at scale is genuinely hard, so systems run periodic self-check diagnostics that pause the job, measure NVLink and compute per node, diagnose, and resume from checkpoint. Imbue open-sourced their bare-metal playbook for a 4,088-H100 cluster: check VBIOS and baseboard firmware, the Mellanox OFED stack, PCIe link speed and width, then run matmuls to measure actual NVLink bandwidth, cordon anything that fails, and swap in a hot spare automatically. That last part is the operating model at scale. You don't fix nodes in the critical path; you drain them and pull from a spare pool, because the cluster is always partially broken and the job has to keep moving.

Fig. 5 · the failure breakdown from Meta's Llama 3 405B run. GPUs and their memory are roughly half; the rest is the long tail an at-scale operator plans around, not against.

paying for it

The economics are why all of the above matters. At $2–6 per GPU-hour, a 16,000-GPU cluster idling during a recovery burns thousands of dollars a minute, and gang scheduling means one bad node can idle the whole job. That's the real argument for the reliability engineering: not uptime for its own sake, but goodput, the useful training throughput net of failures and restarts.

Fig. 6 · the on-demand spread between neoclouds and hyperscalers is wide, and it moves monthly. Big training runs almost never pay on-demand; they live on reserved capacity or capacity blocks.

The pricing spread is wide and moves every month. Neoclouds like Lambda, CoreWeave, and Nebius run an H100 around $2.5–3.5 per hour; AWS lists closer to $6.88. B200s are $5–6 on neoclouds and north of $14 on AWS. Reserved commitments of one to twelve months cut 16–40% off on-demand, and that's where most large training capacity actually lives. Spot is 30–70% cheaper and nearly unusable for gang-scheduled training, because losing any one node preempts the whole synchronous job and reacquiring N contiguous, topology-aligned nodes on spot is a fantasy. That gap is why capacity blocks exist: AWS EC2 Capacity Blocks and GCP's Dynamic Workload Scheduler let you reserve co-located GPUs for a fixed window, booked weeks ahead, because on-demand can't guarantee the topology and reserved is too long a commit for one run. AWS raised those block prices about 15% in early 2026, which tells you which way demand is going.

That's the whole arc of building this stuff. A GPU deployment is not a GPU. It's a dozen layers under one pod, a small network inside one box, and a large one between boxes, and the interesting failures always live in the wiring rather than the silicon. Meta's cluster lost a GPU every three hours and still trained a frontier model, because the whole apparatus around the GPUs (the fabric, the scheduler, the checkpointing, the spare pool) was built to keep moving while parts of it were on fire. That's how you build it. The next part is how you watch it once real traffic arrives, which turns out to be a different problem than watching the GPUs.

One box, eight GPUs, and the wires between them

Harshit Luthra — Thu, 02 Jul 2026 19:13:24 +0000

Originally published at harshit.cloud on 2026-07-04.

We bought two boxes that were supposed to be identical. Eight H100s each, same rack, same image. On one of them a tensor-parallel serve of Llama-3-70B did about 3,000 tokens a second. On the other it did 900, with every GPU pinned at high utilization the whole time. Same model, same code, same card count. The difference was a PCIe switch and a BIOS setting nobody had checked, and it took most of a day to find because every dashboard said both boxes were healthy.

That's the thing about a multi-GPU box. It looks like a bag of eight GPUs. It behaves like a small, opinionated network, and the wiring between the cards matters more than the cards. This is part 2 of the series. Part 1 was the twelve-layer stack under a single GPU pod. This one stays inside one chassis: how the GPUs talk, how to read the topology, why NCCL is slow, and how a 70B model actually lands on the hardware. Part 3 leaves the box.

the box is a network, not a bag of GPUs

The first question about any multi-GPU server is how the GPUs are wired, because that sets a hard ceiling on everything above it. There are two very different animals sold as "8-GPU servers."

An HGX or DGX baseboard wires all eight GPUs through a bank of NVSwitches. Every GPU reaches every other GPU at the full NVLink rate, non-blocking. On H100 and H200 that's NVLink 4 at 900 GB/s per GPU. On B200 it's NVLink 5 at 1.8 TB/s. That flat, full-bandwidth mesh is the reason you can split a model eight ways and have the halves talk fast enough to keep up.

A cheaper "8x PCIe" box has no NVSwitch. The GPUs hang off PCIe switches and the CPU root complexes, and GPU-to-GPU traffic crawls through PCIe, often routed up through the CPU. PCIe Gen5 x16 is about 128 GB/s, Gen4 about 64. NVLink 4 is roughly seven times faster than Gen5 and fourteen times faster than Gen4. That gap is the entire reason tensor parallelism cares about your topology. The two "identical" boxes in the opening weren't identical: one had the NVSwitch mesh, the other routed two of its GPU pairs across a PCIe switch with a BIOS feature quietly strangling them.

One units trap that trips up everyone reading spec sheets: NVIDIA quotes NVLink bandwidth bidirectionally. A100's "600 GB/s" is 300 each way. Pick a convention, state it once, and don't compare someone's unidirectional number to your bidirectional one.

Key Insight: Two servers with the same eight GPUs can differ by 3× on the same job. Before you promise a throughput number, run nvidia-smi topo -m and confirm the GPUs talk over NVLink (the NV-prefixed rows), not over PCIe routed through the CPU (the SYS rows).

Fig. 2 · bandwidth per GPU across the interconnects you'll actually meet. The jump from PCIe to NVLink is the one that decides whether a split model keeps up with itself.

reading nvidia-smi topo -m

You don't have to guess at any of this. nvidia-smi topo -m prints the whole connection matrix, and learning to read it is the single most useful GPU-ops skill after nvidia-smi itself. Every cell tells you how one GPU reaches another, and the symbols form a quality ladder from best to worst:

NV#: connected by # bonded NVLinks. Best. NV18 means eighteen links, full H100 mesh.
PIX: a single PCIe bridge, same switch. Fine.
PXB: multiple PCIe bridges, but not across the CPU host bridge.
PHB: crosses a PCIe host bridge, through the CPU, same NUMA node.
NODE: crosses host bridges within a NUMA node.
SYS: crosses the inter-socket link between CPUs. Worst. This is CPU-to-CPU-to-GPU.

If the GPUs in your tensor-parallel group show NV18 to each other, you're golden. If any pair shows SYS, your collective operations are dragging across the socket interconnect and you've found your 900-tokens-a-second box. The same matrix has a column for GPU-to-NIC affinity, which matters enormously for the multi-node story in part 3: you want the NIC on the same PCIe complex as the GPU it feeds.

NCCL picks a road

Every framework that splits work across GPUs (PyTorch DDP and FSDP, DeepSpeed, Megatron, the tensor-parallel path in vLLM) does its cross-GPU communication through NCCL, NVIDIA's collectives library. NCCL is where "the GPUs need to agree on a number" turns into actual bytes on actual wires, and it auto-picks the road.

Inside one box it prefers, in order: peer-to-peer over NVLink (best), peer-to-peer over PCIe, shared host memory (staged through RAM), then network sockets (worst, and a sign something's misconfigured). When an all-reduce (the step where every GPU merges its numbers with all the others and ends up with the combined result) is slow, the debugging loop is almost always the same handful of moves:

# 1. what did NCCL actually choose?
NCCL_DEBUG=INFO python train.py 2>&1 | grep -iE 'via|transport|channel'
#   "via P2P/direct pointer" = good. "via SHM" or "via NET/Socket" intra-node = bad.

# 2. confirm the GPUs are NVLinked, not routed over SYS
nvidia-smi topo -m

# 3. benchmark against the ceiling
all_reduce_perf -b 8 -e 4G -f 2 -g 8   # from nccl-tests

# 4. if a hang clears when you disable P2P, you have an ACS/IOMMU problem
NCCL_P2P_DISABLE=1 python train.py

The env vars worth knowing are few. NCCL_DEBUG=INFO tells you what topology and transport NCCL chose. NCCL_P2P_LEVEL and NCCL_P2P_DISABLE control peer-to-peer. NCCL_SOCKET_IFNAME picks the bootstrap interface, and pointing it at the wrong one (lo, docker0) is a classic way to make init hang. NCCL_TOPO_FILE lets you hand NCCL a topology description, which you sometimes need on cloud VMs because virtualized PCI hides the real affinity and NCCL guesses wrong. On a healthy 8-GPU NVSwitch box you usually touch none of these and it just works. The trouble starts on anything cheaper or virtualized.

the invisible tax: NUMA and ACS

Two settings below the framework quietly decide whether your bandwidth numbers are real, and neither shows up in a GPU dashboard.

The first is NUMA pinning. A multi-socket server splits its PCIe lanes and RAM between CPU sockets, and each GPU is physically wired to one socket. Run your process on socket 0 while it drives a GPU hung off socket 1, and every host-to-device copy crosses the inter-socket link. NCCL's low-latency protocol stages data through a pinned CPU buffer, so this hits communication, not just data loading. The fix is a one-liner: numactl --cpunodebind=0 --membind=0 <cmd>, matched to the socket that owns the GPU. nvidia-smi topo -m prints the affinity so you know which socket that is. It's genuinely deflating to spend an afternoon profiling and find the answer was a numactl prefix, but that's most of this job.

The second is PCIe ACS, Access Control Services, and it's the one that cost us most of that day. ACS forces PCIe peer-to-peer transactions to route up through the CPU root complex so the platform can police them. That defeats direct GPU-to-GPU DMA across a PCIe switch: latency climbs, throughput collapses, and NCCL can hang outright. ACS has to be off for peer-to-peer to work across a switch. You check it with lspci -vvv | grep -i acsctl and disable it in BIOS or with a setpci loop over the bridges. Its cousin, the IOMMU, does the same routing-through-the-root-complex thing, which is exactly why passthrough GPUs on cloud VMs often show degraded peer-to-peer: the isolation that makes virtualization safe is the isolation that makes GPUDirect slow. If nvidia-smi insists your GPUs "are not P2P capable," the shortlist is ACS enabled, IOMMU on, consumer cards, or GPUs on different root complexes. p2pBandwidthLatencyTest from the CUDA samples confirms which.

fitting a 70B model on the box

The practical question most teams actually have is: how many GPUs does my model need, and how do I split it. Tensor parallelism splits every layer's weight matrices across GPUs, which means every token, at every layer, triggers a collective to recombine the partial results. That makes TP communication-bound and latency-sensitive, which is the real reason it wants NVLink. On a PCIe-only box, tensor parallelism is frequently slower than just pipelining the layers, and vLLM's own guidance says as much: no NVLink, prefer --pipeline-parallel-size over --tensor-parallel-size.

The memory math for Llama-3-70B is a worked example worth carrying around:

Weights: 70B params times 2 bytes for BF16 is about 140 GB. That already doesn't fit on one 80GB card.
KV cache (the model's running memory of the tokens it has already processed): roughly 2.5 GB per sequence at 8K context, climbing to tens of GB at 128K. This is what eats whatever VRAM the weights left behind and sets your max batch size.
Real footprint with activations and framework overhead lands north of 200 GB.

So Llama-3-70B at BF16 wants two H100 80GB cards (--tensor-parallel-size 2), or four A100 40GB, or a fistful of 24GB cards. Tensor parallelism splits the KV cache too, so two GPUs buys you roughly double the batch headroom, not just room for the weights. Quantize to FP8 on Hopper or Blackwell, or INT4 with AWQ, and the weights roughly halve or quarter, which can drop a 70B onto a single card at some quality cost. You can quantize the KV cache too (--kv-cache-dtype=fp8_e5m2), which shrinks the biggest consumer of leftover VRAM once batch sizes climb. And a rule worth internalizing: tensor-parallel a model only when it genuinely doesn't fit one GPU. For a model that does fit, run several replicas at --tensor-parallel-size 1 instead, because TP's per-layer communication is pure overhead you're paying for nothing when one card already holds the whole model. The constraint people forget: your TP size has to divide the number of attention heads evenly, so you don't get to pick arbitrary GPU counts, and on Kubernetes it also has to equal the pod's nvidia.com/gpu limit or the server won't start. (You learn this the moment --tensor-parallel-size 6 refuses to start and the error message does nothing to help.)

the caveats that page you

Dense GPU boxes fail in ways that a single card never does, and most of them present as "one GPU is a little slow" rather than a clean error.

Thermal throttling is first. Eight cards share airflow, and a fully loaded chassis runs hot. Data-center GPUs start clocking down before about 85°C, and nvidia-smi --query-gpu=clocks_throttle_reasons.active will tell you whether it's a software thermal slowdown, a hardware one, or a power cap. Power capping is the sibling: nvidia-smi -pl sets a limit below the card's TDP (700W on an H100 SXM), and a rack without enough power budget will cap every card and slow the whole box uniformly.

Then the straggler problem, which is the nastiest because it hides. Collective operations synchronize every step, so the slowest GPU sets the pace for all of them. One card that's throttling, power-capped, or quietly degraded drags the entire tensor-parallel group, and the symptom is maddening: every GPU shows high utilization, throughput is low, and nothing errors. You find it by looking at per-GPU clocks and temperature and hunting the outlier.

The rest of the list is worth a scan before you sign off on a box:

PCIe lane starvation. A card silently negotiating Gen3 x4 instead of Gen5 x16 is pure mystery slowness. lspci -vv shows the negotiated LnkSta.
NVLink errors. nvidia-smi nvlink -e shows CRC and replay counters. Rising counts mean a flaky link or cable and degraded bandwidth.
Oversubscribed PCIe switch. Cheap boxes put several GPUs behind one switch uplink, and they contend for it.
The /dev/shm trap, if you're on Kubernetes. NCCL stages some intra-node transfers through shared memory, and a container's default 64 MiB /dev/shm hangs multi-GPU serving with no useful error. Mount an emptyDir with medium: Memory at /dev/shm. It's in every production vLLM manifest, and it's why tensor-parallel works on a bare VM but hangs in a pod.
Xid 79, the "fell off the bus" from part 1, shows up here too when a card overheats or loses power delivery under a full load it never saw in acceptance testing.

picking an inference engine

If the box is for serving rather than training, the engine choice matters, and by 2026 the field has settled. All the live engines converge on the same two tricks: continuous batching (decide the batch membership every decode step so the GPU never idles on the slowest request) and paged KV cache (manage the cache like OS virtual memory so you don't waste most of your VRAM on fragmentation). The differences are in the scheduler and the compile strategy.

Fig. 3 · where the four engines land. Start at vLLM; move only when a profiled bottleneck points somewhere specific.

Start with vLLM. It has the broadest model and hardware coverage, it invented PagedAttention, and it stands up with sane defaults faster than anything else. Reach for TensorRT-LLM when a profiled bottleneck justifies the cost, because it delivers the highest raw throughput and lowest latency on NVIDIA hardware but makes you compile a per-model, per-GPU engine and run a heavier ops burden for it. Reach for SGLang when your workload shares prefixes: RAG, multi-turn chat, agents, or high-concurrency MoE, where its RadixAttention prefix cache pulls meaningfully ahead. And don't start new work on TGI; HuggingFace put it in maintenance mode and points people at vLLM or SGLang themselves. One layer up from the engine, NVIDIA's Dynamo wraps any of these for distributed, disaggregated serving with a KV-cache-aware router, and NIM packages them as prebuilt microservices. On a single box the raw engine is what you're tuning, but those are the names you'll meet the moment you scale out.

The two boxes from the opening ran the same vLLM. Same engine, same flags, one served three times the traffic. The engine was never the variable. The wires were. Which is the whole lesson of a single node, and also the reason part 3 is about the wires between nodes, where the same story plays out at a hundred times the scale and a hundred times the cost.

The dozen layers under a GPU pod

Harshit Luthra — Thu, 02 Jul 2026 19:13:19 +0000

Originally published at harshit.cloud on 2026-07-02.

The pod was stuck in Pending and the node had eight H100s sitting idle. kubectl describe node said nvidia.com/gpu: 0. nvidia-smi on the host printed all eight cards, healthy, 40°C, nothing running. So the hardware was fine, the driver was fine, and Kubernetes was convinced there were zero GPUs in a box that cost more than my car.

That gap is the whole job. A GPU pod doesn't run on a GPU. It runs on about a dozen layers stacked between the silicon and your container, and any one of them can be quietly broken while every layer above and below it looks green. If you've shipped normal apps on Kubernetes, a GPU pod looks identical right up until it doesn't: underneath sits a stack of hardware and driver pieces a web pod never touches, and that's where the surprises live. This series is about running those layers in production without getting paged. Part 1 is the anatomy: what the layers are, what breaks at each one, and which numbers actually tell you the truth. Part 2 is a single box with eight GPUs and the wires between them. Part 3 is scaling past one box, where the network becomes the machine. Part 4 is watching the whole thing under real traffic, part 5 is scaling it to zero when nobody's using it without the bill or the cold start eating you, part 6 is the routing layer in front of it all, where the right load balancer buys a 2× speedup for free, part 7 keeps it breathing through loads and deploys, and part 8 is sharing it with other teams without the tenants, or a security boundary that isn't where you think, burning you.

the dozen layers between silicon and your pod

Start at the bottom and climb. Each layer trusts the one under it and lies to the one above it when things go wrong.

The silicon is the GPU itself plus the NVSwitch fabric that wires the GPUs on a board together, plus the NIC (a ConnectX-7 or BlueField-3) that carries traffic off the box. This is where hardware faults live: ECC errors (bit-flips in memory the card catches and corrects), thermal throttle, a card that stops answering on the PCIe bus.

Firmware sits on the card. Modern GPUs have a GSP, a GPU System Processor, a little RISC-V core that runs firmware and offloads work the host driver used to do. When you hear about a GPU "hanging" for no visible reason, the GSP firmware timing out is a common culprit.

The kernel driver is nvidia.ko and friends (nvidia-uvm, nvidia-peermem). It's a kernel module, which means it's compiled against your exact kernel headers. Upgrade the kernel without rebuilding the module and the driver won't load. This is the layer that breaks on a Tuesday because someone patched the base image.

The userspace driver is libcuda.so, the CUDA driver API. It ships with the driver, not with CUDA, and this trips people up constantly. nvidia-smi talks to the driver through NVML, which is why nvidia-smi can work while your actual CUDA program fails: they're using different entry points into the same stack.

The CUDA runtime is libcudart plus the math and collective libraries: cuBLAS, cuDNN, NCCL. Here's the thing nobody tells you on day one. PyTorch and TensorFlow wheels bundle their own copy of all of this. When you pip install torch==2.x+cu128, you are installing CUDA 12.8, cuDNN, and NCCL inside the wheel. The host node doesn't need a CUDA install at all. It needs a driver new enough to satisfy that bundled runtime, and nothing more. Once that clicks, half the version confusion evaporates.

The NVIDIA Container Toolkit (nvidia-ctk, libnvidia-container, currently 1.19.1) is the bridge between the host driver and the container. At container start it injects /dev/nvidia* and the driver libraries into the container's filesystem. Miss this and you get the classic symptom: nvidia-smi works on the host, fails inside the pod.

The container runtime is containerd or CRI-O running runc underneath. It has to be told to use the nvidia runtime. If default_runtime_name in /etc/containerd/config.toml isn't set, pods land on the node with no GPU access and no obvious error. (Recent GPU Operator versions wire this through the NRI/CDI plugin instead of a default runtime, but the failure mode is identical: get it wrong and the pod sees no GPU.)

The device plugin (0.19.x) is the piece that talks to the kubelet. It counts the GPUs and advertises them as nvidia.com/gpu: 8, or nvidia.com/mig-1g.10gb: 56 if you're slicing. When this crashes or can't reach the driver, you get nvidia.com/gpu: 0 and pods stuck Pending. That was my incident at the top. The device plugin had crash-looped after a driver-container restart and never re-registered.

Node Feature Discovery and GPU Feature Discovery label the node with what it has: GPU model, memory, compute capability, MIG profile, driver version. The scheduler reads those labels to place pods. Wrong labels, wrong placement.

The GPU Operator (v26.3.x) is the thing that installs and manages every layer above the kernel as a set of DaemonSets. It runs the driver as a container, wires the toolkit, deploys the device plugin, DCGM, the MIG manager. It's a huge convenience and one more control loop to debug when it gets stuck reconciling.

On top, the scheduler (kube-scheduler, or Kueue, Volcano, Run:ai, or Slurm) decides which pod lands on which GPU. And finally the workload: a training job that needs all N GPUs at once or nothing, or an inference server that would happily take a seventh of one card.

Twelve layers. The reason GPU infra feels harder than normal infra isn't any single layer. It's that the failure at layer 3 shows up as a symptom at layer 9, and the tooling at layer 9 has no idea layer 3 exists.

Key Insight: The layer that breaks and the symptom you notice are rarely the same layer. The pod is Pending up at the scheduler, but the cause is a driver that didn't load down near the metal. When a GPU pod misbehaves, debug from the bottom of the stack up, not the top down.

One part of this stack is quietly being rebuilt under you. The device plugin advertising nvidia.com/gpu: 8 is a flat count: a pod asks for a number and gets whatever GPUs the node has. Kubernetes 1.34 (September 2025) made Dynamic Resource Allocation (DRA) generally available, and it's the eventual replacement for that model. DRA is a ResourceClaim API, the way a PersistentVolumeClaim is for storage, so a pod can ask for "two NVLink-connected GPUs with at least 40GB each" instead of a bare count, and it's how rack-scale multi-node NVLink (GB200 NVL72) gets scheduled at all. NVIDIA ships a DRA driver through the GPU Operator. The device plugin is still the common path in mid-2026, but a stack diagram drawn today should treat it as the model DRA is replacing, not the permanent one.

the version matrix that pages you

The single most common self-inflicted outage is a version mismatch, and the matrix has four axes: driver, CUDA toolkit, cuDNN, and framework. The good news is that three compatibility mechanisms mean you rarely have to line up all four exactly. The bad news is that nobody explains which mechanism they're relying on, so it feels like luck.

Backward compatibility is the easy one. A newer driver runs older CUDA binaries, always. An R580 driver runs a CUDA 12 app and a CUDA 13 app without complaint. So keeping the driver ahead of everything is safe.

CUDA minor-version compatibility is the one you lean on daily. Any CUDA 12.x toolkit runs on any driver that supports 12.0. A driver from the 525 era will run a CUDA 12.8 binary, because they share the CUDA 12 major. This is why the framework-bundles-its-own-CUDA pattern works: the wheel carries CUDA 12.8, your node has some 12-capable driver, and they meet in the middle.

Forward compatibility is the escape hatch. The cuda-compat package ships updated libcuda stubs that let a newer CUDA major run on an older driver branch. It's how you run a CUDA 13 app on a node still pinned to an R535 driver you can't upgrade yet. It only works on data-center GPUs, and it's a deliberate override, not something to build on.

Here's the current data-center driver picture as of mid-2026:

Branch	Type	EOL	CUDA
R535	LTS	June 2026	12
R580	LTS	June 2028	13
R595	Production	March 2027	13
R610	New Feature	Aug 2026	13

If you take one thing from this section: manage the driver, let the framework carry the rest. Pin your training and serving images to NGC base images (nvcr.io/nvidia/pytorch:25.xx) where NVIDIA has already matched CUDA, cuDNN, and NCCL for you, and you delete an entire class of 2am pages.

One current gotcha worth flagging if you're on Blackwell. Drivers from 580.65.06 turn on Coherent Driver Memory Management by default for GB200 and GH200 on Kubernetes, and CDMM is incompatible with both MIG and GPUDirect Storage right now. If you buy GB200s planning to slice them with MIG, check that first, because it's not going to be on the datasheet.

slicing one GPU three ways

A single H100 has 80GB of HBM (its fast on-board memory) and enough compute to serve dozens of small models. Handing a whole card to a workload that uses 8% of it is how you set money on fire. There are three ways to share a GPU, and they are not interchangeable. The difference is isolation.

Fig. 2 · the same GPU shared three ways. Isolation drops as you move right; utilization convenience goes up. Pick by how much you trust the tenants.

MIG (Multi-Instance GPU) is hardware partitioning, available on the data-center cards (A100, A30, H100, H200, B200) but not the smaller inference GPUs like the L4 or A10G, which fall back to MPS. It cuts one GPU into up to seven instances, each with its own SMs (the GPU's compute cores), its own dedicated slice of HBM, its own memory controller and L2. A fault in one instance doesn't touch the others. That's real hardware isolation, the kind you want when tenants don't trust each other. The profiles are named [compute]g.[memory]gb. On an H100 80GB you get seven 1g.10gb slices, or two 3g.40gb, or one 7g.80gb that's just the whole card back. An H200's 141GB gives you seven 1g.18gb slices; a B200's 180GB gives seven 1g.23gb. The quirk that catches everyone: compute fractions go 1/7, 2/7, 3/7, 4/7, 7/7. There is no 5g or 6g profile. Memory is quantized in eighths. So a 1g slice gets one-seventh of the compute but one-eighth of the memory, and the arithmetic never quite lines up.

MPS (Multi-Process Service) is a daemon that multiplexes several processes into one GPU context so their kernels run genuinely concurrently, not round-robin. You can cap each client's memory (CUDA_MPS_PINNED_DEVICE_MEM_LIMIT) and compute share (CUDA_MPS_ACTIVE_THREAD_PERCENTAGE). What you don't get is hardware memory protection or clean fault isolation. One client that OOMs can take its neighbors down with it. MPS is for high-throughput inference where you own all the tenants and want better SM utilization than time-slicing gives you.

Time-slicing is the crude one. The device plugin just lies about the GPU count, advertising one physical card as ten replicas. Ten pods land, and they take turns via context switching. There is no memory partitioning and no fault isolation at all. If one pod grabs 70GB of an 80GB card, the other nine OOM. It's fine for notebooks, CI, and dev clusters where the work is bursty and nobody's SLA depends on it. It has no business in front of production traffic.

	Memory isolation	Fault isolation	Concurrency	Use for
MIG	hardware	hardware	spatial	multi-tenant, untrusted, prod serving
MPS	soft caps	weak	true concurrent	trusted high-throughput inference
Time-slicing	none	none	round-robin	dev, notebooks, CI

You can stack them: partition a card into seven MIG slices, then time-slice each slice for burstier workloads. Most teams don't need that. Most teams need to notice they're running one 8% workload per $30k card and switch to 7× 1g.10gb.

the metrics that lie to you

Here's the number that ruins more capacity planning than any other: GPU-Util. When nvidia-smi shows 95% and everyone relaxes, they've misread it. DCGM_FI_DEV_GPU_UTIL answers exactly one question: was a kernel running during the sample window. It says nothing about how much of the silicon that kernel used. You can see 95% GPU utilization and 25% of the actual compute capacity in use at the same moment, and both numbers are honest about different things. Somebody will still screenshot the 95% into a capacity deck, and now you're being asked to buy more of a card you're already wasting.

Fig. 3 · GPU-Util says a kernel ran. MFU says how much of the chip did useful work. The gap between them is memory stalls, collective communication, and non-matmul overhead you paid for anyway.

The number that matters for training is MFU, Model FLOPs Utilization: the FLOPs your model actually did divided by the theoretical peak. It's the metric the Llama and PaLM papers report, and 35–50% is considered excellent for large-scale training. The gap between "a kernel ran" and "the chip did useful math" is memory-bandwidth stalls, NCCL all-reduce waiting on the network, attention softmax and layernorm that aren't matmuls, optimizer steps, and activation recompute. All of it counts against your wall clock. None of it counts as useful FLOPs.

The money follows directly. On an 8-GPU H100 node at roughly $3 per GPU-hour, the difference between 25% and 45% MFU is about 1.8× the effective cost per token. Sticker price per GPU-hour is the number vendors compete on. Utilization is the number that actually sets your bill.

For real telemetry, DCGM (Data Center GPU Manager) is the layer, and dcgm-exporter scrapes it into Prometheus. The fields worth putting on a dashboard from day one:

DCGM_FI_PROF_PIPE_TENSOR_ACTIVE: Tensor Core utilization, the one that tracks real training throughput.
DCGM_FI_PROF_DRAM_ACTIVE: HBM bandwidth in use. High here with low tensor activity means you're memory-bound.
DCGM_FI_DEV_FB_USED: HBM used. Your OOM early-warning.
DCGM_FI_DEV_XID_ERRORS: the last Xid code. The single most important reliability signal on the box.
DCGM_FI_DEV_ECC_DBE_VOL and the row-remap fields: memory health, trending toward RMA.
DCGM_FI_DEV_CLOCK_THROTTLE_REASONS: a bitmask telling you whether the card is throttling on power, thermals, or a reliability limit.

dcgmi diag -r 3 runs about twelve minutes of escalating health checks (memory bandwidth, PCIe, NVLink, thermals under load) and is the thing to run before you trust a node you just recovered.

the Xid codes worth memorizing

Xid errors are the driver's way of telling the kernel log that the hardware had to correct or retry something it shouldn't have. They land in dmesg as NVRM: Xid (PCI:0000:xx:00): <code>. A nonzero Xid is not always fatal, but it's never nothing. A handful are worth knowing on sight because they change what you do next.

Fig. 4 · the Xid triage most on-call runbooks converge on. The split that matters is app-fault versus hardware-fault, because one restarts a pod and the other cordons a node.

Xid 13, 31, 43 are usually your fault, not the hardware's: illegal memory access, a bad kernel, a page fault from application code. Restart the workload, look at the model, not the card.

Xid 48 is a double-bit ECC error, uncorrectable. Xid 94 is a contained ECC error, where the damage stayed inside the offending app and the other apps on the card survived. Xid 95 is the uncontained version, where the blast radius crossed apps, and the GPU has to be reset before anything restarts on it. The 94/95 split is the one to internalize: contained means drain politely, uncontained means the card is compromised until reset.

Xid 63 and 64 are row-remapping events. Modern HBM can retire bad memory rows the way a disk retires bad sectors. A 63 is the card recording that it did this; persistent 64s (remap failures) mean it's running out of spare rows and it's an RMA candidate. Watch the trend, don't panic on the first one.

Xid 79 is the one that ruins a night: "GPU has fallen off the bus." The card stopped answering on PCIe entirely. Thermal, power delivery, seating, a dying board. The node needs a reset, and if the same physical slot throws it again, that's a card headed back for RMA. Field reports put it around 3% of H100 deployments in the first year, which sounds small until you multiply it by a thousand-GPU fleet.

Xid 119 and 120 are the GSP firmware timing out. Reset the GPU. On a few driver versions it's common enough that ops teams disable the GSP firmware as a workaround, which tells you how much fun that particular bug is.

The remediation ladder most teams settle on is boring and effective: app restart or driver reload or node reboot clears roughly 60% of incidents within fifteen minutes; anything that survives that gets dcgmi diag -r 3; anything that fails the diag gets cordoned and sent back.

what to actually care about

If you're standing up GPU infrastructure and wondering where to spend your attention, the honest ranking isn't the one the marketing implies. It's roughly this. Get the driver-and-toolkit layer boringly stable, because that's where the self-inflicted outages live. Instrument MFU and the Xid stream before you instrument anything pretty, because those two tell you whether you're wasting money and whether the hardware is dying. Decide your sharing model (MIG for multi-tenant, whole cards for training) before you have tenants, because retrofitting isolation is miserable. And treat the twelve-layer stack as the thing it is: a place where a green dashboard at layer 9 can sit directly on top of a card that fell off the bus at layer 1.

This is also usually the point where teams bring in someone who has already burned a few weeks on Xid codes and MIG partitioning, rather than doing it live on their own GPU bill. GPU and ML infrastructure builds are part of the independent infrastructure consulting work I take on.

The pod that was stuck Pending at the top of this post came back the moment the device plugin re-registered. Fifteen seconds of fix, forty minutes of staring at a healthy nvidia-smi wondering how the machine could see eight GPUs that Kubernetes swore didn't exist. That distance, between what the hardware knows and what the scheduler believes, is where most of your on-call rotation lives too. The next part goes inside a single box with eight of these cards and the wires that decide whether they cooperate or just sit next to each other.

The git commands I actually run every day

Harshit Luthra — Wed, 20 May 2026 19:10:05 +0000

Originally published at harshit.cloud on 2026-05-20.

I've been using git for a decade and most of what I type still fits on a single hand. The 200-page Pro Git book is wonderful and almost none of it survives contact with a real Tuesday. What survives is a small, boring set of commands that get rerun constantly.

This post is that list, ordered by how often my fingers actually type them. Aliases are from the oh-my-zsh git plugin (enabled in most zsh configs that exist); the full command sits next to the alias so it's portable.

the daily eight

These are the ones I'd type in my sleep. If you're not using all eight already, picking them up pays back inside a week.

gst

git status

gst

I run this between every other command. It's the cheapest sanity check git has. Branch, ahead/behind, staged, unstaged, untracked. Two seconds. If you only learn one alias, learn this one.

glola

git log --oneline --graph --decorate --all

glola | head -30

The one true log. Graph of every branch (local + remote), one line per commit, colored refs. Pipe through head because most of the time you only care about the last 20-30 commits.

gd / gds

git diff / git diff --staged

gd          # what's changed but not staged
gds         # what's staged and about to be committed

gds before every commit. If you set delta as your pager (brew install git-delta, then pager = delta in ~/.gitconfig), the output stops being painful to read.

gcam

git commit -a -m

gcam "fix: trailing slash in webhook URL"

Quick one-line commits for small fixes. For anything bigger I drop the -m and let $EDITOR open so I can write a proper message with a body.

gpsup

git push --set-upstream origin <current-branch>

gpsup

First push of a new branch. The full command is annoying to type, so gpsup figures out the current branch name itself. After the first push, plain gp (just git push) works because upstream is set.

gco / gcb

git checkout / git checkout -b

gco main             # switch to main
gco -                # switch to previous branch
gcb feature/login    # create + switch to new branch

gco - is the one to notice. Like cd - for branches. When you're bouncing between two branches all day, it's a single keystroke each way instead of typing the name.

gpf

git push --force-with-lease

gpf

After rebasing or amending. Always use --force-with-lease, never --force. The lease version refuses to push if someone else has pushed to your branch since your last fetch, saving you from silently overwriting a teammate's work. There is no good reason to ever type --force in 2026.

gfa

git fetch --all --prune

gfa

Refresh every remote, prune deleted remote branches. Run before you start anything that depends on knowing the current state of the world. The --prune half is what makes the cleanup ritual below work.

checkout recent branches

git branch lists alphabetically, which is useless. What you actually want is "that branch from Tuesday," which means sorting by last commit:

git config --global alias.recent \
  "for-each-ref --sort=-committerdate refs/heads/ \
   --format='%(HEAD) %(color:yellow)%(refname:short)%(color:reset) \
             %(color:green)(%(committerdate:relative))%(color:reset) %(contents:subject)'"

git recent | head -10

That covers looking. For switching, pipe the same list into fzf and you never type a branch name again:

# fco: fuzzy-checkout a recent branch
fco() {
  local branch
  branch=$(git for-each-ref --sort=-committerdate refs/heads/ \
             --format='%(refname:short)' \
           | fzf --height 40% --reverse \
                 --preview 'git log --oneline --decorate --color=always -15 {}')
  [ -n "$branch" ] && git checkout "$branch"
}

Branches arrive sorted by recency, so the one you want is almost always in the top three. Type two letters of its name, Enter, done. The preview pane shows the branch's recent commits so you can confirm it's the right Tuesday. gco - still wins for bouncing between exactly two branches; fco wins for everything else. (brew install fzf if you don't have it. You want it for Ctrl-R history search anyway.)

the cleanup ritual

Run this weekly. If you've ever scrolled through 80 stale branches looking for the one you actually want, you already know why.

The easy half deletes every local branch whose tip is already in main:

gfa
git branch --merged main | grep -v '\*\|main\|master' | xargs -n1 git branch -d

Works only if your team uses merge commits. Most don't. GitHub's "Squash and merge" creates a brand-new commit on main with a different SHA, so git branch --merged never catches your local branch. Its commits aren't in main's history at all.

The workaround: after gfa, any branch whose tracked remote was deleted shows as [gone]. Those are usually your merged-and-deleted PRs.

Usually, not always. [gone] only means the remote tracking branch is gone. Nearly always that's a squash-merged PR whose branch GitHub auto-deleted. But it can also be a branch you pushed, someone deleted server-side, and you never merged. So don't force-delete every [gone] branch with git branch -D. I once watched one show [gone] while it still held 26 unmerged commits; a force-delete there loses them for good.

So check each [gone] branch for patch-equivalence against the base before deleting. Squash-merges get caught, genuinely unmerged work gets kept. This lives in my ~/.gitconfig as git gone:

# ~/.gitconfig, under [alias]  →  run as: git gone
gone = "!f() { \
    git fetch --all --prune; \
    base=$(git rev-parse --abbrev-ref origin/HEAD 2>/dev/null); base=${base:-origin/main}; \
    for b in $(git for-each-ref --format='%(refname:short) %(upstream:track)' refs/heads \
               | awk '$2==\"[gone]\"{print $1}'); do \
      if [ -z \"$(git cherry \"$base\" \"$b\" | grep '^+')\" ]; then git branch -D \"$b\"; \
      else echo \"kept $b (commits not in $base)\"; fi; \
    done; }; f"

One command does the whole ritual: the git fetch --all --prune prunes the dead remote refs, then the loop deletes the merged local branches in the same pass. No separate gfa first.

git cherry compares by patch-id, not SHA. A squash-merged branch shows every commit as - (an equivalent already exists in the base) and gets deleted; a branch with real unpushed work shows + lines and stays. The -D is only reached after patch-equivalence is proven, so it never eats unmerged work.

Or install git-trim (brew install git-trim), which does the same classification and more. It catches squash-merges even when the tracking ref isn't [gone], and skips diverged branches by default.

git trim                # dry-run
git trim --confirm      # actually delete

This is the closest thing to "did my PR ship?" you can ask git directly.

the weekly four

Not in your fingers yet, but should be.

`git switch` and `git restore`

git switch -c new-feature           # create + switch
git restore --staged file.txt       # unstage
git restore --source=abc123 file.go # restore single file from any commit

switch and restore split the four jobs checkout used to do. The one I reach for most is restore --source=<sha> <path>. Translation: "grab this single file from three commits ago without touching anything else."

interactive rebase with autosquash

git commit --fixup=abc123       # fixup commit targeting abc123
# ... keep working ...
git rebase -i --autosquash main # all fixups slot into place automatically

The single biggest workflow win I've found in ten years of git. While reviewing your own PR you find a bug four commits back. Don't fix it on top. --fixup=<sha> creates a commit targeting the offender, and the autosquash rebase squashes everything into place when you're done. Install git-absorb (brew install git-absorb) and it even picks the target SHA for you: edit the files, run git absorb --and-rebase, done.

`git reflog`, the universal undo

git reflog
git reset --hard HEAD@{5}

Every change to HEAD is logged for 90 days. Bad rebase? reflog. Deleted branch? reflog. There is almost nothing in git you can't undo if you know about it.

`git worktree`

git worktree add ../proj-hotfix hotfix/prod-down
git worktree remove ../proj-hotfix

Need to fix a prod bug while halfway through a feature? Don't stash. worktree add gives you a second checkout in a sibling directory, sharing the same .git. I use it constantly for "let me review your PR" without leaving my own branch.

set it once

Five config lines and a daemon. Enable, forget.

git config --global rerere.enabled true          # remember conflict resolutions, replay them
git config --global push.default current         # `git push` pushes current branch to same name
git config --global push.autoSetupRemote true    # first push sets upstream automatically
git config --global diff.algorithm histogram     # cleaner diffs than the default myers
git config --global merge.conflictStyle zdiff3   # conflict markers include the common ancestor
git maintenance start                            # background gc/prefetch on a schedule

autoSetupRemote retires gpsup entirely. zdiff3 shows the original code both sides diverged from; once you've used it, plain <<<<<<< markers feel like flying blind.

when something is broken

Not daily, but when the question is "when did this start," nothing else answers it:

git log -S "functionName"          # pickaxe: commits where this string was added or removed
git blame -w -C -C -C file.go      # blame the logic's actual author, not the formatter
git log -p --follow file.go        # full file history, including across renames
git range-diff @{u} @              # what a rebase actually changed; run before force-pushing

-S searches the content of the diff, not commit messages. Different thing entirely from --grep. And plain blame gives credit to whoever last ran Prettier; -w -C -C -C follows the code across whitespace changes, moves, and file boundaries to the person who wrote the logic.

the four tools worth installing today

fzf (brew install fzf). Powers the fco branch picker above, plus fuzzy Ctrl-R history.
git-absorb (brew install git-absorb). Auto-fixup commits without picking SHAs.
delta (brew install git-delta). Diff and blame output that doesn't hurt to look at.
lazygit (brew install lazygit). TUI for the operations that are tedious on CLI: partial commits, stash management, conflict resolution.

This post used to end with two AI shell helpers for the stuff git can't tell you; those now live in their own TIL.

ten years in, the surprise

After a decade, the command I run most isn't commit. It isn't push. It's gst, hundreds of times a day, between every other operation. The most-used git command in my shell is the one that does nothing.