<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: praveen9161</title>
    <description>The latest articles on DEV Community by praveen9161 (@praveen9161).</description>
    <link>https://dev.to/praveen9161</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4250535%2Fec6cc154-dae6-4a3c-b3c3-fec382d37d73.png</url>
      <title>DEV Community: praveen9161</title>
      <link>https://dev.to/praveen9161</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/praveen9161"/>
    <language>en</language>
    <item>
      <title>A slow GPU isn't a broken GPU: grading GPUs per workload in Kubernetes (measured on real A10Gs)</title>
      <dc:creator>praveen9161</dc:creator>
      <pubDate>Tue, 17 Nov 2026 20:13:02 +0000</pubDate>
      <link>https://dev.to/praveen9161/a-slow-gpu-isnt-a-broken-gpu-grading-gpus-per-workload-in-kubernetes-measured-on-real-a10gs-4mij</link>
      <guid>https://dev.to/praveen9161/a-slow-gpu-isnt-a-broken-gpu-grading-gpus-per-workload-in-kubernetes-measured-on-real-a10gs-4mij</guid>
      <description>&lt;p&gt;GPU clusters treat a GPU as healthy or broken. Real GPUs also end up &lt;strong&gt;slow&lt;/strong&gt;: a lowered&lt;br&gt;
power limit, a clock lock, heat. That middle state is expensive in a specific way: in&lt;br&gt;
synchronous distributed training every step waits for the slowest GPU, so one slow GPU&lt;br&gt;
slows &lt;em&gt;every&lt;/em&gt; GPU in the job, and you pay for all of them.&lt;/p&gt;

&lt;p&gt;The usual answer is to remove the slow GPU. But "slow" depends on the work. So I built&lt;br&gt;
&lt;strong&gt;GPUFit&lt;/strong&gt;, an open-source tool that grades every GPU &lt;em&gt;per workload&lt;/em&gt; (training, fine-tuning,&lt;br&gt;
inference) and uses that grade for scheduling in Kubernetes: a degraded GPU is kept out of&lt;br&gt;
synchronous training and keeps serving inference.&lt;/p&gt;

&lt;p&gt;Everything below was measured on real NVIDIA GPUs on AWS. Total spend: about &lt;strong&gt;$8.50&lt;/strong&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  The same fault hurts workloads very differently
&lt;/h2&gt;

&lt;p&gt;One A10G, power limit lowered from its default 300 W, three runs each:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Power limit&lt;/th&gt;
&lt;th&gt;Heavy matrix multiply (training-like)&lt;/th&gt;
&lt;th&gt;Memory copy (inference-like)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;250 W&lt;/td&gt;
&lt;td&gt;0.94&lt;/td&gt;
&lt;td&gt;1.00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;150 W&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0.68&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;1.00&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;100 W&lt;/td&gt;
&lt;td&gt;0.35&lt;/td&gt;
&lt;td&gt;1.00&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;At 150 W the GPU lost a third of its speed for compute-heavy work and nothing for&lt;br&gt;
memory-bound work. A "healthy/unhealthy" flag can't express that. A per-workload grade can.&lt;/p&gt;

&lt;h2&gt;
  
  
  How GPUFit works
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;An &lt;strong&gt;agent&lt;/strong&gt; on each GPU node turns GPU health (from &lt;code&gt;nvidia-smi&lt;/code&gt;) into a score per
workload. Training uses the &lt;em&gt;weakest&lt;/em&gt; component, because one straggler sets the pace.&lt;/li&gt;
&lt;li&gt;A &lt;strong&gt;controller&lt;/strong&gt; moves a node with a GPU that's unfit for training into an &lt;strong&gt;inference
pool&lt;/strong&gt;, using Kubernetes &lt;strong&gt;DRA device taints&lt;/strong&gt; (Dynamic Resource Allocation, stable in
Kubernetes 1.37). Training claims don't tolerate the taint; inference claims do. No custom
scheduler, and it works with any DRA driver, including NVIDIA's.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Hysteresis&lt;/strong&gt;: a GPU must stay degraded for a while before it moves, and stay healthy for
longer before it comes back, so noisy GPUs don't flap.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Observe mode&lt;/strong&gt; (the default) changes nothing and reports what it &lt;em&gt;would&lt;/em&gt; do, plus
&lt;code&gt;kubectl gpufit report&lt;/code&gt;: training time on degraded GPUs and the estimated waste.
&lt;strong&gt;Enforce mode&lt;/strong&gt; applies it. Optional &lt;strong&gt;rescue&lt;/strong&gt; moves running training jobs that opted
in off a degraded GPU; they resume from their checkpoint.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;kubectl gpufit why &amp;lt;pod&amp;gt;&lt;/code&gt; explains a Pending GPU pod. The scheduler says "cannot allocate
all claims" whether the cluster is busy or the GPUs are unfit; this tells them apart.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Install: &lt;code&gt;helm install gpufit oci://ghcr.io/praveen9161/charts/gpufit --version 0.1.0 -n gpufit --create-namespace&lt;/code&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Result 1: -25% cost per useful GPU-hour
&lt;/h2&gt;

&lt;p&gt;Two A10Gs, k3s, NVIDIA's DRA driver. GPU B's power limit cut to 150 W.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Without GPUFit&lt;/strong&gt;, Kubernetes put a 2-GPU training job on A+B. It ran at &lt;strong&gt;0.84&lt;/strong&gt; of its
healthy speed: one slow GPU dragged the healthy one down.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;With GPUFit&lt;/strong&gt;, B moved to the inference pool. Training got the healthy GPU (1.00) and
B served inference at &lt;strong&gt;0.98&lt;/strong&gt; of a healthy GPU.&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Useful work (healthy-GPU equivalents)&lt;/th&gt;
&lt;th&gt;Efficiency&lt;/th&gt;
&lt;th&gt;$ per useful GPU-hour&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Without GPUFit&lt;/td&gt;
&lt;td&gt;1.49&lt;/td&gt;
&lt;td&gt;74%&lt;/td&gt;
&lt;td&gt;$1.35&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;With GPUFit&lt;/td&gt;
&lt;td&gt;1.98&lt;/td&gt;
&lt;td&gt;99%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;$1.02&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The honest caveat: with only two GPUs and no spare, the training job itself got slower&lt;br&gt;
(one GPU instead of two). The gain is that almost no paid GPU time was wasted. In a bigger&lt;br&gt;
cluster, training would get a healthy spare.&lt;/p&gt;

&lt;h2&gt;
  
  
  Result 2: rescuing a job that's already running
&lt;/h2&gt;

&lt;p&gt;Four A10Gs, GPUFit installed with its Helm chart. A 2-GPU training job (400 steps,&lt;br&gt;
checkpoint every 50); at step 100 one of its GPUs drops to 150 W.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Run&lt;/th&gt;
&lt;th&gt;Job time&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Healthy&lt;/td&gt;
&lt;td&gt;901 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Degraded, GPUFit in observe mode (job stays as a straggler)&lt;/td&gt;
&lt;td&gt;1001 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Degraded, GPUFit in enforce + rescue mode&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;987 s&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;What the rescue did: 32 s after the power cut the node left the training pool; 36 s later&lt;br&gt;
(a 30 s grace period) GPUFit evicted the job; 22 s later both pods ran again on healthy GPUs&lt;br&gt;
and resumed from step 100.&lt;/p&gt;

&lt;p&gt;A 14-second win is small, and that's the interesting part. The straggler lost 13% of every&lt;br&gt;
remaining second; the rescue cost about 86 s, fixed. So rescue pays off once more than&lt;br&gt;
&lt;strong&gt;about 10 minutes&lt;/strong&gt; of work remain after the GPU degrades (an estimate from these&lt;br&gt;
measurements; about 16 minutes with production timers). Real training jobs run for hours.&lt;/p&gt;

&lt;h2&gt;
  
  
  What didn't work (yet)
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The waste estimate is too pessimistic.&lt;/strong&gt; GPUFit's report estimates speed from the score
(score 50 = half speed). The real job ran at 0.87, so the estimate was &lt;strong&gt;3.8x&lt;/strong&gt; the measured
waste. Scores from hardware settings are a trigger, not a speedometer.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Small scale.&lt;/strong&gt; 2-4 GPUs, synthetic workloads, one kind of induced fault. A straggler slows
every GPU in its job, so the loss grows with job size; I haven't measured beyond 2-GPU jobs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Network slowdowns aren't covered.&lt;/strong&gt; In the FALCON study (Alibaba, HKUST),
network slowdowns were 7.2x more common than GPU compute slowdowns.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Things that bit me
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;k3s' default pod network (10.42.0.0/16) overlapped my VPC (also 10.42.0.0/16). Nodes
silently routed each other's real addresses into the pod network.&lt;/li&gt;
&lt;li&gt;NVIDIA's DRA kubelet plugin only runs on nodes labelled &lt;code&gt;nvidia.com/gpu.present=true&lt;/code&gt;,
normally set by GPU feature discovery, which I hadn't installed: zero GPUs published.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;helm uninstall&lt;/code&gt; left my taints behind, so training would have avoided those GPUs forever.
The fix (a pre-delete hook) first failed too: the controller was still running and
re-applied the taint one second after cleanup. Now the hook stops the controller first.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;helm upgrade --reuse-values&lt;/code&gt; turns a &lt;code&gt;null&lt;/code&gt; value override into an empty string. My agent
quietly stopped scheduling on the test cluster.&lt;/li&gt;
&lt;li&gt;An evicted pod still reports &lt;code&gt;Running&lt;/code&gt; while it shuts down. My first rescue timeline said
"running again after 0 s".&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  How this compares
&lt;/h2&gt;

&lt;p&gt;Others detect slow GPUs well: NVIDIA's nvidia-resiliency-ext scores each GPU inside&lt;br&gt;
training, ByteDance's MegaScale and FALCON (Alibaba, HKUST) find stragglers across thousands of GPUs,&lt;br&gt;
NVIDIA's DRA driver taints GPUs with hard faults, and HeaRank (KDD 2026) steers GPUs likely&lt;br&gt;
to fail toward interruption-tolerant work. What I couldn't find in the open is the next step:&lt;br&gt;
grading by measured &lt;em&gt;performance per workload&lt;/em&gt; and keeping the slow GPUs earning on&lt;br&gt;
inference through standard Kubernetes DRA, with the cost effect measured.&lt;/p&gt;

&lt;h2&gt;
  
  
  What's next
&lt;/h2&gt;

&lt;p&gt;Use the real per-GPU speed from the training jobs (via NVIDIA's own tooling) instead of&lt;br&gt;
estimating it, grade GPUs separately for LLM prefill and decode so degraded GPUs can feed&lt;br&gt;
decode pools, catch network slowdowns, and measure at 8-32 GPUs.&lt;/p&gt;

&lt;p&gt;Code, every raw result, and a cost calculator for your own fleet:&lt;br&gt;
&lt;strong&gt;&lt;a href="https://github.com/praveen9161/gpufit" rel="noopener noreferrer"&gt;https://github.com/praveen9161/gpufit&lt;/a&gt;&lt;/strong&gt; (Apache-2.0).&lt;br&gt;
Calculator: &lt;a href="https://praveen9161.github.io/gpufit/calculator/" rel="noopener noreferrer"&gt;https://praveen9161.github.io/gpufit/calculator/&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Feedback welcome, especially if you run GPU fleets: how often do your GPUs end up slow but not broken?&lt;br&gt;
&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxsks7eun515v4d057d6g.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxsks7eun515v4d057d6g.png" alt=" " width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

</description>
      <category>kubernetes</category>
      <category>gpu</category>
      <category>mlops</category>
      <category>devops</category>
    </item>
  </channel>
</rss>
