<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Andy White</title>
    <description>The latest articles on DEV Community by Andy White (@andy_white_e5f5d48052c6ab).</description>
    <link>https://dev.to/andy_white_e5f5d48052c6ab</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3046591%2Fb1c93555-756c-472c-9efc-ca7c0f8d436b.jpg</url>
      <title>DEV Community: Andy White</title>
      <link>https://dev.to/andy_white_e5f5d48052c6ab</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/andy_white_e5f5d48052c6ab"/>
    <language>en</language>
    <item>
      <title>I ran the same QLoRA 7B job 8 times on Vast.ai RTX 4090s. The bill went from $0.19 to $0.81. Here is what moved it.</title>
      <dc:creator>Andy White</dc:creator>
      <pubDate>Thu, 12 Nov 2026 19:06:33 +0000</pubDate>
      <link>https://dev.to/andy_white_e5f5d48052c6ab/i-ran-the-same-qlora-7b-job-8-times-on-vastai-rtx-4090s-the-bill-went-from-019-to-081-here-586d</link>
      <guid>https://dev.to/andy_white_e5f5d48052c6ab/i-ran-the-same-qlora-7b-job-8-times-on-vastai-rtx-4090s-the-bill-went-from-019-to-081-here-586d</guid>
      <description>&lt;p&gt;When people compare a rented RTX 4090 with a cloud A100, they multiply the hourly GPU price by the training time. I wanted to see&lt;br&gt;
what actually shows up on the Vast.ai invoice for one fixed job on different hosts.&lt;/p&gt;

&lt;p&gt;The job:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Qwen2.5-7B-Instruct, 4-bit QLoRA, &lt;code&gt;yahma/alpaca-cleaned&lt;/code&gt;, 300 steps, checkpoint every 50 steps.&lt;/li&gt;
&lt;li&gt;Image &lt;code&gt;pytorch/pytorch:2.5.1-cuda12.4-cudnn9-runtime&lt;/code&gt;, dependencies installed at start.&lt;/li&gt;
&lt;li&gt;One RTX 4090, host reliability &amp;gt;= 0.98, network &amp;gt;= 500 Mbit/s.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Three things from the invoices:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Traffic: $0.81 against $0.19.&lt;/strong&gt; On the first run the GPU hour looked cheap, but the bill was $0.81: $0.13 for the GPU and
$0.66 for traffic, because the host charged for bandwidth and the job downloads a 15 GB model. After I started picking hosts by
the full hourly price including expected traffic, the next run was billed $0.19 (0.44 h on the machine, about 19 minutes of it
training).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Same GPU, different bill: $0.21 to $0.38.&lt;/strong&gt; On 4 November I ran the job six more times on six different 4090 hosts, with no
failures. The most expensive run was on a host that trained at about 6 s per step instead of about 4, so training took 31
minutes instead of 17–19. I did not measure why that host was slower.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Start-up is billed too.&lt;/strong&gt; In those six runs the first training step came 4.5 to 8.5 minutes after renting. You pay for the
image pull, the pip install and the model download.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;For scale: the $0.19 run would be about $1.35–1.80 for the same job on one A100 at AWS on-demand rates. That is one job; I would not&lt;br&gt;
generalise it.&lt;/p&gt;

&lt;p&gt;How do you filter out slow 4090 hosts before renting? And do you bake everything into one large image to cut the start-up, or cache&lt;br&gt;
the weights somewhere?&lt;/p&gt;

&lt;p&gt;Disclosure: these runs were launched by a small supervisor service I build (NomadGPU), which is why I have the per-run billing.&lt;br&gt;
Happy to share the logs.&lt;/p&gt;

</description>
      <category>cloud</category>
      <category>infrastructure</category>
      <category>llm</category>
      <category>machinelearning</category>
    </item>
    <item>
      <title>Fine-tuning on rented GPUs: how to lose a host without losing your steps</title>
      <dc:creator>Andy White</dc:creator>
      <pubDate>Wed, 11 Nov 2026 13:53:33 +0000</pubDate>
      <link>https://dev.to/andy_white_e5f5d48052c6ab/fine-tuning-on-rented-gpus-how-to-lose-a-host-without-losing-your-steps-4lno</link>
      <guid>https://dev.to/andy_white_e5f5d48052c6ab/fine-tuning-on-rented-gpus-how-to-lose-a-host-without-losing-your-steps-4lno</guid>
      <description>&lt;p&gt;GPU marketplaces like Vast.ai are a cheap way to fine-tune open models, as long as the data is not confidential: the host can see&lt;br&gt;
the machine while your job runs. The harder problem is that training scripts assume the hardware under them stays alive.&lt;/p&gt;

&lt;p&gt;When a host disappears, two things break:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Checkpoints that only exist on the local disk are gone.&lt;/li&gt;
&lt;li&gt;If the machine hangs instead of dying, the rental keeps billing while nothing trains.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Here is what I ended up building, and what the measurements were.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. Upload checkpoints outside the training loop.&lt;/strong&gt; A small agent in the container watches the checkpoint directory and uploads&lt;br&gt;
each finished checkpoint to Cloudflare R2. The training script writes a marker file when a checkpoint is complete, so half-written&lt;br&gt;
ones are never uploaded. The agent uses a temporary key that only works for this job's folder.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Count traffic and start-up, not only the GPU price.&lt;/strong&gt; One QLoRA 7B run was billed $0.81, most of it traffic for downloading&lt;br&gt;
the model; after choosing hosts by the full hourly price including traffic, the next run was $0.19. In six later runs the first&lt;br&gt;
training step came 4.5 to 8.5 minutes after renting, and those minutes are billed. My own rule of thumb, not a measurement:&lt;br&gt;
checkpoint often enough that one lost interval costs less than one start-up.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Handle the dead host.&lt;/strong&gt; A supervisor outside the GPU machine checks the instance and the agent's heartbeat. When the host&lt;br&gt;
is gone it rents another machine, the agent restores the last checkpoint and the script resumes. In a test where I destroyed the&lt;br&gt;
machine twice during one run, training continued after 3.5 and 13 minutes.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Handle the hung host.&lt;/strong&gt; If the GPU stops responding for five minutes, or the job makes no progress for half an hour while the&lt;br&gt;
GPU sits idle, the job is moved to another machine.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. Handle the dead supervisor.&lt;/strong&gt; If the supervisor itself goes away, nobody replaces failed machines or checks spending&lt;br&gt;
limits. So the agent watches how long it has gone without a new storage key from the supervisor and, after a safety window, stops&lt;br&gt;
the job and the machine from the inside. I tested this on Vast with the window shortened to a few minutes: the machine stopped,&lt;br&gt;
and after that Vast billed only for disk. In normal settings the window is about 2.5–3.5 hours; I have not run a test of that&lt;br&gt;
length.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Do it yourself or not.&lt;/strong&gt; If your team has someone to run it, dstack and SkyPilot are good open-source tools for this, and you&lt;br&gt;
probably don't need a service.&lt;/p&gt;

&lt;p&gt;Disclosure: I run NomadGPU (&lt;a href="https://nomadgpu.com" rel="noopener noreferrer"&gt;nomadgpu.com&lt;/a&gt;), a small service that does the above on your own Vast.ai&lt;br&gt;
account, with spending limits in dollars. Live numbers from our own test jobs, a small sample:&lt;br&gt;
&lt;a href="https://status.nomadgpu.com/s/status" rel="noopener noreferrer"&gt;status.nomadgpu.com/s/status&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>automation</category>
      <category>cloud</category>
      <category>infrastructure</category>
      <category>machinelearning</category>
    </item>
    <item>
      <title>LoRA on spot RTX 4090s: what happens when the host dies mid-run.</title>
      <dc:creator>Andy White</dc:creator>
      <pubDate>Tue, 10 Nov 2026 16:13:33 +0000</pubDate>
      <link>https://dev.to/andy_white_e5f5d48052c6ab/lora-on-spot-rtx-4090s-what-happens-when-the-host-dies-mid-run-gmd</link>
      <guid>https://dev.to/andy_white_e5f5d48052c6ab/lora-on-spot-rtx-4090s-what-happens-when-the-host-dies-mid-run-gmd</guid>
      <description>&lt;p&gt;Community GPU marketplaces like Vast.ai rent an RTX 4090 for well under a dollar an hour. For a lot of fine-tuning&lt;br&gt;
work that is the cheapest compute you can get. The catch is that the machine belongs to someone else: hosts reboot,&lt;br&gt;
lose their network, get outbid, or simply disappear in the middle of your run.&lt;/p&gt;

&lt;p&gt;I wanted to know two things: how much cheaper a real fine-tune is on such a machine, and what it takes for that&lt;br&gt;
fine-tune to survive the host disappearing.&lt;/p&gt;

&lt;h3&gt;
  
  
  The job
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;QLoRA fine-tune of Qwen2.5-7B-Instruct (4-bit), yahma/alpaca-cleaned, 300 steps, checkpoint every 50 steps.&lt;/li&gt;
&lt;li&gt;Public image &lt;code&gt;pytorch/pytorch:2.5.1-cuda12.4-cudnn9-runtime&lt;/code&gt;, dependencies installed at start.&lt;/li&gt;
&lt;li&gt;One RTX 4090, host reliability ≥ 0.98, network ≥ 500 Mbit/s.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  What it cost
&lt;/h3&gt;

&lt;p&gt;The number below is taken from the Vast.ai invoice for one run without failures, not from "price × time".&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Time on the machine&lt;/td&gt;
&lt;td&gt;0.44 h (about 19 min of training, the rest is start-up and the model download)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Billed by Vast.ai&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;≈ $0.19&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Same 0.44 h on one A100 at AWS on-demand rates (p4d, per GPU)&lt;/td&gt;
&lt;td&gt;≈ $1.80&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;If the A100 trained ~1.5× faster&lt;/td&gt;
&lt;td&gt;≈ $1.35&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;So &lt;strong&gt;86–89% cheaper on this job&lt;/strong&gt;. Not "90% on everything": other jobs, idle time and moves between machines change&lt;br&gt;
the number. The summary is &lt;a href="https://nomadgpu.com/#numbers" rel="noopener noreferrer"&gt;on the landing page&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Six later runs of the same job on 4 November, on six different hosts and without failures, were billed $0.21–0.38 by&lt;br&gt;
Vast. The most expensive one ran on a host that trained at about 6 s per step instead of about 4, so training took&lt;br&gt;
31 minutes instead of 17–19.&lt;/p&gt;

&lt;p&gt;One thing that surprised me: on the first attempt the bill was $0.81, not $0.19. The GPU hour was cheap, but the host&lt;br&gt;
charged for traffic, and downloading a 15 GB model ate most of the savings. Since then the host is chosen by the full&lt;br&gt;
hourly price &lt;strong&gt;including traffic&lt;/strong&gt;, not by the GPU price alone.&lt;/p&gt;

&lt;h3&gt;
  
  
  Killing the machine on purpose
&lt;/h3&gt;

&lt;p&gt;Then the interesting part. Halfway through training I destroyed the instance by hand.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://nomadgpu.com/demo.mp4" rel="noopener noreferrer"&gt;Watch the recording, about 60 s&lt;/a&gt;: training → instance destroyed → new machine → "restored from checkpoint" → training continues.&lt;/p&gt;

&lt;p&gt;What happened:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Within about half a minute the supervisor noticed the agent had stopped reporting and marked the host as lost.&lt;/li&gt;
&lt;li&gt;It rented a replacement with the same requirements.&lt;/li&gt;
&lt;li&gt;The new machine downloaded the image and the model (this is where most of the ~5 minutes of downtime went).&lt;/li&gt;
&lt;li&gt;Training continued &lt;strong&gt;from checkpoint 100&lt;/strong&gt;, not from step 0. Nothing to do on my side; I got two Telegram
messages — "host lost" and "running again".&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;In a separate test I destroyed the machine twice during training: the job continued after 3.5 and 13 minutes.&lt;/p&gt;

&lt;h3&gt;
  
  
  How it works
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;An agent in the container&lt;/strong&gt; reports a heartbeat every minute and uploads checkpoints to object storage
(Cloudflare R2) every few minutes, as soon as the training script marks one ready.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A supervisor&lt;/strong&gt; outside the GPU machine watches all jobs. No heartbeat, host gone, GPU not responding, or no
progress for too long — it rents another machine and the job resumes from the last checkpoint.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Hard spending limits&lt;/strong&gt; per job, day, week and month: warning at 80%, at 100% the machine is released and progress
is kept.&lt;/li&gt;
&lt;li&gt;Each job gets a &lt;strong&gt;temporary storage key&lt;/strong&gt; that only works for its own folder; checkpoints can be encrypted inside
the container before upload.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Live numbers, not just one demo
&lt;/h3&gt;

&lt;p&gt;One staged failure proves little, so the supervisor publishes its own numbers on a&lt;br&gt;
&lt;a href="https://status.nomadgpu.com/s/status" rel="noopener noreferrer"&gt;public status page&lt;/a&gt;. On 9 November — on our own test jobs, so a small sample —&lt;br&gt;
it showed 15 job interruptions over 30 days and 12 jobs resumed, with a median of about one minute and a 90th percentile&lt;br&gt;
of eight minutes until the job was back on a machine. The other three were my own test jobs, cancelled before the&lt;br&gt;
replacement came up.&lt;/p&gt;

&lt;h3&gt;
  
  
  What it does not solve
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The host can see your data while the job runs.&lt;/strong&gt; That is true of every GPU marketplace. Don't run personal or
confidential data this way; a data-center-only mode reduces the risk but does not remove it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Start-up is slower than a big cloud:&lt;/strong&gt; every new machine downloads the image and the model. In six runs on
4 November the first training step came 4.5–8.5 minutes after renting, and those minutes are billed.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One GPU per job&lt;/strong&gt; for now — no multi-node training.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Your code is still yours:&lt;/strong&gt; if the script crashes because of a bug, moving to another machine won't fix it.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Try it
&lt;/h3&gt;

&lt;p&gt;I packaged this as NomadGPU: it runs on your own Vast.ai account (you pay Vast directly), and the first 14 days are&lt;br&gt;
free. If you have a fine-tune that keeps dying with its host, I'm happy to set it up with you:&lt;br&gt;
&lt;a href="https://nomadgpu.com" rel="noopener noreferrer"&gt;nomadgpu.com&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Questions about the method or the numbers — ask below, I'll answer with the logs.&lt;/p&gt;

</description>
      <category>machinelearning</category>
      <category>gpu</category>
      <category>llm</category>
    </item>
  </channel>
</rss>
