TL;DR: We use one local command to search for a GPU, start an instance, wait for SSH, upload a queue, run its processing script, copy results home, verify the transfer, and destroy the instance. A detached watchdog sets a spending deadline in case the controller crashes. The script below is a starting point: supply the model image, an offer query, and a queue-specific process.sh. Keep a separate backup of valuable results before destroying the instance.
Why we put the steps in one command
We are a small content team that rents GPUs on Vast.ai for AI video work. Earlier in this series, we covered individual model runs and the setup they needed. The recurring problem was everything around inference: finding a suitable machine, pulling weights, moving inputs and outputs, and remembering to clean up.
Setup can dominate a short run. In one LatentSync self-test, the queued batch took 15.9 minutes, including 11 minutes of setup. Running more jobs in one session made that setup easier to justify. It also made cleanup more consequential: a stopped instance still billed for storage.
Our pipeline has two rules. Results must pass a local count and size check before normal cleanup. Separately, a watchdog must be able to destroy the instance even if the controller disappears. Those rules can conflict at the deadline, so the deadline is an emergency cap, not a promise that unfinished work will be saved.
The contract for a batch
The local queue is a directory containing inputs and an executable process.sh. The script in this article copies that directory to /workspace/queue and runs its process.sh on the rented machine. That script must put every deliverable under /workspace/results. It is where model-specific commands belong.
Prepare an image with the runtime and repository your workload needs. We built a public Docker image on GHCR using GitHub Actions, then used an on-start script to download only the selected weights. For example, DL_GROUPS=h3 selected the H3 weights, and the Vast on-start command was bash /workspace/bootstrap/onstart.sh. The image build took about 11 minutes in our setup.
The pipeline deliberately treats process.sh as a black box. An image-to-video queue and a lip-sync queue need different arguments, but they need the same rental, transfer, verification, and cleanup sequence.
| Step | What the controller checks |
|---|---|
| Search | Offer is rentable, reliable, has enough disk, and has no inbound traffic charge |
| Create | Instance starts with the selected image, disk, label, and on-start command |
| Connect | SSH accepts a command before any upload begins |
| Process |
process.sh exits successfully and writes to /workspace/results
|
| Copy | Results arrive in the local output directory |
| Verify | Local file count and total bytes match the remote directory |
| Destroy | The instance is destroyed and the current instance list confirms it |
Choose the offer before starting the clock
The CLI search we used is vastai search offers '<query>' -o 'dph+' --raw. Filtering matters more than selecting the first low hourly price. We look for reliability of at least 0.97, sufficient disk, and inet_down_cost of zero. We also check cuda_max_good or driver_version against the image. One A100 we found in Sweden had driver 535, which did not fit the software we needed.
Provision disk generously when pulling large weights. We use at least 150 GB for those sessions; 40 GB is suitable only for smaller ones. On an earlier RTX PRO 6000 Blackwell test, inbound traffic for 245 GB of model weights added $0.64. The GPU portion was $1.47, but the total was $2.18. That is why the query in the example excludes inbound traffic charges.
Read the offer returned by search before supplying the hourly rate to the controller. Its cost deadline is calculated from your stated rate, so a wrong rate makes the deadline wrong too. It is a guardrail, not accounting software: it cannot predict every charge. If comparing this approach with another vendor’s API, check that vendor’s current pricing page.
A controller and a detached watchdog
Save the following as batch.py. It uses the verified Vast CLI commands for offer search, creation, transfer, and destruction. The CLI should already be configured with your key. Set VAST_API_KEY in the environment for the separate v1 instance-list check; the script never prints it.
The watchdog is another invocation of the same file, started as a detached process. It waits until the deadline, tries destruction up to three times, then checks the v1 instance list. On a normal run, the controller destroys the instance after verification. A later watchdog check will find nothing to do.
import argparse
import json
import os
import shlex
import subprocess
import sys
import time
import urllib.request
from pathlib import Path
from urllib.parse import urlsplit
API = "https://console.vast.ai/api/v1/instances/"
def vast(*args, check=True, input_text=None):
return subprocess.run(
["vastai", *map(str, args)],
text=True, input=input_text, capture_output=True, check=check
)
def listed(instance_id):
key = os.environ["VAST_API_KEY"]
request = urllib.request.Request(
API, headers={"Authorization": f"Bearer {key}"}
)
with urllib.request.urlopen(request, timeout=20) as response:
payload = json.load(response)
items = payload.get("instances", []) if isinstance(payload, dict) else payload
return any(str(item.get("id")) == str(instance_id) for item in items)
def destroy(instance_id):
for attempt in range(3):
if not listed(instance_id):
return True
vast("destroy", "instance", instance_id,
check=False, input_text="y\n")
time.sleep(5)
return not listed(instance_id)
def ssh_prefix(instance_id):
value = vast("ssh-url", instance_id).stdout.strip()
if value.startswith("ssh://"):
url = urlsplit(value)
return ["ssh", "-p", str(url.port),
f"{url.username}@{url.hostname}"]
parts = shlex.split(value)
if parts and parts[0] == "ssh":
return parts
raise RuntimeError(f"Unrecognized ssh-url output: {value}")
def remote(prefix, command, check=True):
return subprocess.run(
prefix + [command], text=True,
capture_output=True, check=check
)
def manifest(directory):
files = [p for p in directory.rglob("*") if p.is_file()]
return len(files), sum(p.stat().st_size for p in files)
def watchdog(instance_id, deadline):
while time.time() < deadline:
time.sleep(min(20, max(1, deadline - time.time())))
if not destroy(instance_id):
raise RuntimeError("Destroy was not confirmed")
def main(args):
queue = Path(args.queue).resolve()
output = Path(args.output).resolve()
if not (queue / "process.sh").is_file():
raise RuntimeError("Queue needs a process.sh")
if output.exists() and any(output.iterdir()):
raise RuntimeError("Output directory must be empty")
if args.cap_usd <= 0 or args.hourly_rate <= 0:
raise RuntimeError("Cap and hourly rate must be positive")
offers = json.loads(vast(
"search", "offers", args.query, "-o", "dph+", "--raw"
).stdout)
offer = next(
item for item in offers
if item["rentable"]
and item["reliability"] >= 0.97
and item["disk_space"] >= args.disk
and item["inet_down_cost"] == 0
)
created = json.loads(vast(
"create", "instance", offer["id"],
"--image", args.image, "--disk", args.disk,
"--label", args.label, "--onstart", args.onstart,
"--env", f"-e DL_GROUPS={args.dl_groups}",
"--ssh", "--direct", "--raw"
).stdout)
instance_id = created["new_contract"]
deadline = time.time() + 3600 * args.cap_usd / args.hourly_rate
subprocess.Popen(
[sys.executable, str(Path(__file__).resolve()),
"--watchdog", str(instance_id), str(deadline)],
start_new_session=True,
stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL
)
print(f"Instance {instance_id}; deadline set")
prefix = None
while time.time() < deadline:
try:
prefix = ssh_prefix(instance_id)
if remote(prefix, "true", check=False).returncode == 0:
break
except (RuntimeError, subprocess.CalledProcessError):
pass
time.sleep(10)
else:
raise RuntimeError("SSH did not become ready before deadline")
remote(prefix, "mkdir -p /workspace/queue /workspace/results")
vast("copy", str(queue) + "/", f"{instance_id}:/workspace/queue/")
remote(prefix, "bash /workspace/queue/process.sh")
output.mkdir(parents=True, exist_ok=True)
vast("copy", f"{instance_id}:/workspace/results/", str(output) + "/")
count = remote(
prefix,
"python3 -c 'import pathlib,json; p=pathlib.Path("
"\"/workspace/results\"); f=[x for x in p.rglob(\"*\") "
"if x.is_file()]; print(json.dumps([len(f), "
"sum(x.stat().st_size for x in f)]))'"
)
expected = tuple(json.loads(count.stdout))
actual = manifest(output)
if expected[0] == 0 or actual != expected:
raise RuntimeError(f"Transfer mismatch: remote={expected}, local={actual}")
print(f"Verified {actual[0]} files, {actual[1]} bytes")
if not destroy(instance_id):
raise RuntimeError("Results copied, but destroy was not confirmed")
print("Instance destroyed and absence confirmed")
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("--watchdog", nargs=2)
parser.add_argument("--queue")
parser.add_argument("--output")
parser.add_argument("--query")
parser.add_argument("--image")
parser.add_argument("--label", default="video-batch")
parser.add_argument("--onstart", default="onstart.sh")
parser.add_argument("--dl-groups", default="h3")
parser.add_argument("--disk", type=int, default=150)
parser.add_argument("--cap-usd", type=float)
parser.add_argument("--hourly-rate", type=float)
args = parser.parse_args()
if args.watchdog:
watchdog(int(args.watchdog[0]), float(args.watchdog[1]))
else:
main(args)
The script expects JSON output with an offer list and a creation response containing new_contract. Check those shapes against your installed CLI before relying on an unattended run. It also assumes the rented image has python3 for the remote count and size check. The watchdog inherits VAST_API_KEY, so keep that variable available until cleanup is confirmed.
A call can then be reduced to one command. Replace the image and offer query with values you inspected, and set --hourly-rate to the rate you used when calculating the cap:
python3 batch.py \
--queue ./queue \
--output ./results \
--query 'reliability >= 0.97 num_gpus = 1 disk_space >= 150 inet_down_cost = 0' \
--image YOUR_IMAGE \
--label h3-batch \
--onstart onstart.sh \
--dl-groups h3 \
--disk 150 \
--cap-usd 1 \
--hourly-rate YOUR_OFFER_HOURLY_RATE
For a first run, use a small queue and watch the output. The command starts the rental immediately; the cap begins at creation, so weight downloads and SSH startup consume part of it. Increase the cap only after allowing for setup and the work you have actually queued.
Verify, back up, then destroy
The local comparison catches incomplete transfers by checking file count and total bytes. It does not prove that two same-sized files have identical contents. For valuable output, keep a separate backup before destruction. In our workflow, we copied results to a cloud remote with rclone and checked its file count and size; one backup check matched 5,342 files and 1.379 GiB against local storage.
We learned the cleanup rule the expensive way. A stopped instance with a 90 GB disk kept billing about $0.0167 per hour, or about $0.40 per day. We left one stopped for days, the balance went negative, the stopped instance could not be restarted, and we never downloaded the trained result. Now “finished” means results are copied, backed up, and the instance is destroyed. We verify absence through the v1 instances endpoint because an older v0 listing returned an error that looked like “0 instances.”
The code sends y to vastai destroy instance, matching the CLI prompt we encountered. Newer CLI versions also offer -y. If processing fails, the controller raises an error and leaves recovery time until the watchdog deadline; inspect the instance promptly. Once that deadline arrives, the watchdog prioritizes the cap.
What it cost us
On a Vast A100 SXM4 80GB in Czechia, MiniMax H3 image-to-video took about 86 seconds per five-second clip. The GPU rate was $1.19 per hour, about $1.25 with disk, and the run cost about $0.03 per clip. A session producing 82 hero clips cost $3.59 over about three hours, including setup; 72 GB of weights downloaded in roughly eight minutes.
Our earlier RTX PRO 6000 Blackwell test ran for 56 minutes and cost $2.18: $1.47 for GPU time and $0.64 for inbound traffic on 245 GB of weights. A LatentSync 1.6 run on a Vast RTX 3090 at $0.27 per hour with free inbound took 238 seconds for one clip, about $0.09. These are our runs, not promises about future offers.
Conclusion
A batch script does not make rented GPU work effortless. It gives each session a repeatable beginning and a definite end. We still inspect the offer, match its driver to the image, size the disk for the weights, and keep model commands inside the queue’s processing script. The payoff is that setup serves a batch, outputs are checked before cleanup, and a controller failure has a deadline.
If you want to try the same approach, this is our Vast.ai referral link. Start with a small queue, confirm that your CLI and image match the skeleton, and make destruction part of the successful path.
Top comments (0)