Before You Destroy a Rented GPU, Prove Your Results Are Safe

TL;DR We rent GPUs on Vast.ai for AI video work. Our rule now is simple: copy every result off the instance, make a second copy in cloud storage, compare file counts and byte totals, then verify content before destroying the instance. A stopped instance still bills for storage. We learned that after leaving one stopped for days; the balance went negative, we could not restart it, and we never downloaded the trained result.

Why “the job finished” is not the finish line

A render finishing on a rented GPU means the files exist on that machine. It does not mean we have them.

This distinction matters when a session produces many clips. Our MiniMax H3 session produced 82 hero clips. The GPU work was useful only if those clips made it back to our workspace. The same applies to model weights, training outputs, logs, and any intermediate files we might need to diagnose a failed run.

We once stopped an instance while a trained result was still on its disk. Storage continued billing at about $0.0167 per hour, or about $0.40 per day for a 90 GB disk. We left it stopped for days. The balance went negative, a stopped instance could not be started with that balance, and the result was never downloaded.

That mistake changed our shutdown sequence. “Stop” is not our archival step. We copy, verify, back up, and then destroy.

Give every session its own output directory

Before starting a batch, we choose one directory on the rented machine for final results. On the receiving machine, we give that session its own local directory and its own cloud backup prefix.

A dedicated prefix makes verification much easier. If an old file is already in the cloud destination, a file count might look reassuring while hiding a missing new file. It also lets us tell exactly which session a backup belongs to.

For a new Vast.ai account, this is our referral link. Check current offer details before renting: disk capacity and inbound traffic charges can change the cost of a session. We have seen inbound traffic billed separately when downloading large model weights.

Keep the output directory focused on files you intend to retain. Temporary caches and downloaded weights belong elsewhere unless you explicitly want to archive them. For large weights, we provision disk generously; our H3 setup downloaded 72 GB of weights.

Copy results back with rsync

We use an SSH connection to pull results to a local machine. vastai ssh-url ID provides the connection details for an instance. Put the host, port, and user from that connection into environment variables, then run the transfer from the receiving machine:

export INSTANCE_ID="your-instance-id"
export SSH_USER="root"
export SSH_HOST="your-instance-host"
export SSH_PORT="your-instance-ssh-port"
export LOCAL_DIR="$PWD/results/session-a"

vastai ssh-url "$INSTANCE_ID"

mkdir -p "$LOCAL_DIR"

rsync -a --partial \
  -e "ssh -p ${SSH_PORT}" \
  "${SSH_USER}@${SSH_HOST}:/workspace/outputs/" \
  "$LOCAL_DIR/"

The trailing slash on /workspace/outputs/ means “copy the contents of this directory.” The local directory is a session-specific destination. --partial helps when a connection drops during a large transfer, and rerunning rsync is a practical way to resume the copy.

The same general rule applies if you use vastai copy SRC DST: confirm the destination contains the expected files before treating the transfer as complete. A successful command is useful evidence, but we still check the files themselves.

Do not destroy the instance just because rsync exited successfully. The local copy is only the first checkpoint.

Make a SHA-256 manifest

A manifest records the expected digest of every file in the local results directory. We keep it outside that directory so it does not accidentally become part of the result set.

The script below walks regular files, rejects symbolic links, and writes paths relative to the session directory. It also prints the file count and total bytes. It rejects filenames containing newlines because they make a plain text checksum manifest harder to inspect safely.

# save as make_manifest.py
import hashlib
import pathlib
import sys

root = pathlib.Path(sys.argv[1]).resolve()
manifest = pathlib.Path(sys.argv[2]).resolve()

if not root.is_dir():
    raise SystemExit(f"Missing results directory: {root}")

files = sorted(path for path in root.rglob("*") if path.is_file())

if any(path.is_symlink() for path in root.rglob("*")):
    raise SystemExit("Results contain a symbolic link; inspect it first")

total_bytes = 0

with manifest.open("w", encoding="utf-8") as output:
    for path in files:
        relative = path.relative_to(root).as_posix()
        if "\n" in relative or "\r" in relative:
            raise SystemExit(f"Unsupported filename: {relative!r}")

        digest = hashlib.sha256()
        with path.open("rb") as source:
            for chunk in iter(lambda: source.read(1024 * 1024), b""):
                digest.update(chunk)

        output.write(f"{digest.hexdigest()}  {relative}\n")
        total_bytes += path.stat().st_size

print(f"files={len(files)} bytes={total_bytes}")

For example, run it with python3 make_manifest.py "$LOCAL_DIR" "$PWD/SHA256SUMS.session-a". Save the manifest alongside your local backup records. If a file changes later, its digest changes too.

A manifest made after the local transfer cannot prove what existed on the rented machine before the transfer. It gives us a clear record of what we received and a basis for checking later copies. If the source application reports an expected output count, compare that count with the local result as well.

Back up to cloud storage and compare the copies

We use rclone to copy the local results to a cloud remote, then upload the manifest. The remote path below should be a fresh prefix for this session:

export REMOTE_DIR="your-remote:gpu-results/session-a"
export MANIFEST="$PWD/SHA256SUMS.session-a"

python3 make_manifest.py "$LOCAL_DIR" "$MANIFEST"

rclone copy "$LOCAL_DIR" "$REMOTE_DIR"
rclone copyto "$MANIFEST" "$REMOTE_DIR/SHA256SUMS"

Our verification has three parts:

1. Compare the names and sizes of the local result files with a recursive listing of the cloud prefix.
2. Compare file contents, downloading remote data for the check where needed.
3. Download the cloud copy of the manifest and compare it with the local manifest.

Here is a compact pre-destroy check to run after the upload. It assumes LOCAL_DIR, REMOTE_DIR, and MANIFEST are set as above.

#!/usr/bin/env bash
set -euo pipefail

: "${LOCAL_DIR:?Set LOCAL_DIR}"
: "${REMOTE_DIR:?Set REMOTE_DIR}"
: "${MANIFEST:?Set MANIFEST}"

listing="$(mktemp)"
remote_manifest="$(mktemp)"
trap 'rm -f "$listing" "$remote_manifest"' EXIT

rclone lsjson "$REMOTE_DIR" --recursive --files-only > "$listing"

python3 - "$LOCAL_DIR" "$listing" <<'PY'
import json
import pathlib
import sys

root = pathlib.Path(sys.argv[1])
listing = json.loads(pathlib.Path(sys.argv[2]).read_text())

local = {
    path.relative_to(root).as_posix(): path.stat().st_size
    for path in root.rglob("*")
    if path.is_file()
}

remote = {
    item["Path"]: item["Size"]
    for item in listing
    if item["Path"] != "SHA256SUMS"
}

if local != remote:
    missing = sorted(local.keys() - remote.keys())
    extra = sorted(remote.keys() - local.keys())
    changed = sorted(
        name for name in local.keys() & remote.keys()
        if local[name] != remote[name]
    )
    raise SystemExit(
        f"Backup mismatch: missing={missing}, extra={extra}, "
        f"different_size={changed}"
    )

print(f"Matched files={len(local)} bytes={sum(local.values())}")
PY

rclone check --download "$LOCAL_DIR" "$REMOTE_DIR" --exclude SHA256SUMS
rclone copyto "$REMOTE_DIR/SHA256SUMS" "$remote_manifest"
cmp "$MANIFEST" "$remote_manifest"

printf '%s\n' "Local results and cloud backup verified."

The name and size comparison catches missing, extra, and truncated files quickly. rclone check --download adds a content comparison; it may transfer substantial data, so allow time for it. The downloaded manifest confirms that the cloud prefix also contains the exact checksum record we generated locally.

In one backup verification, we matched 5,342 files and 1.379 GiB between local storage and the cloud remote. Those were the checks we wanted to see before removing the rented disk.

Our pre-destroy checklist, in the order we run it:

Step	Evidence we want
rsync from the instance	Exit code 0 and the expected files locally
SHA-256 manifest	File count and byte total recorded
rclone copy to a fresh prefix	Listing matches names and sizes
rclone check --download	Contents match
Manifest compared in the cloud	cmp reports no difference

The last check happens after destroy

Once the local copy, cloud copy, and manifest checks pass, we destroy the instance. The CLI prompts for confirmation; this form supplies the answer for a scripted shutdown:

echo y | vastai destroy instance "$INSTANCE_ID"

We then verify that the instance is gone. An old v0 instances listing was deprecated and once returned an error that looked like “0 instances,” so we do not treat that response as proof. We check the v1 instances endpoint at https://console.vast.ai/api/v1/instances/ with an Authorization: Bearer <key> header, keeping the key in an environment variable or a protected file rather than printing or hard-coding it.

We also use a cost cap and a detached deadline watchdog during unattended sessions. The watchdog retries destroy if the controller crashes. It is a guardrail for the rental meter, not a replacement for copying results promptly.

What it cost us

Our H3 batch on a Vast A100 SXM4 80GB in Czechia produced 82 hero clips during a roughly three-hour session for $3.59, including setup. A separate LatentSync 1.6 queued batch self-test took 15.9 minutes and cost $0.07; setup accounted for about 11 minutes. The painful cost was the stopped 90 GB disk: about $0.0167 per hour, or $0.40 per day, followed by a trained result we never recovered. These are our runs, not price guarantees.

Batching work makes GPU rental economical for us because setup takes a meaningful share of a short session. Safe shutdown is part of that economics. We treat the local copy, the cloud copy, and their verification as deliverables of the job. Only then is the instance finished.