DEV Community: CloudGPU

The hourly GPU price is not your agent's task budget

CloudGPU — Sun, 15 Nov 2026 12:59:50 +0000

An agent can find a cheap GPU and still make an expensive decision. The quoted hourly rate is only one input: billing increments, environment preparation, model download time, output persistence and uncertain startup results change what a task actually costs.

Here is a small, provider-neutral way to make that decision inspectable before an agent is allowed to spend.

1. Represent the billing increment explicitly

Two offers can advertise the same hourly rate but bill differently. At an illustrative $0.40/hour, a 12-minute task costs $0.08 on a source that bills whole minutes, and $0.40 on one with a one-hour minimum and whole-hour increments. An early stop does not erase the agreed minimum.

Use decimal arithmetic for money, and keep the minimum separate from the increment:

from decimal import Decimal, ROUND_CEILING

def estimate_compute_fee(seconds, hourly_usd, minimum_seconds, increment_seconds):
    if seconds <= 0 or minimum_seconds <= 0 or increment_seconds <= 0:
        raise ValueError("Durations and increments must be positive")
    rate = Decimal(hourly_usd)
    if not rate.is_finite() or rate < 0:
        raise ValueError("Rate must be finite and nonnegative")
    duration = Decimal(str(seconds))
    units = (duration / Decimal(increment_seconds)).to_integral_value(
        rounding=ROUND_CEILING
    )
    billed_seconds = max(Decimal(minimum_seconds), units * increment_seconds)
    return rate * billed_seconds / Decimal(3600)

print(estimate_compute_fee(720, "0.40", 60, 60))      # 0.08
print(estimate_compute_fee(720, "0.40", 3600, 3600))  # 0.40

This is an estimate, not an invoice. Production integrations also need the provider's actual start condition, pricing expiry and finalized ledger. Do not round away the precision the ledger uses.

2. Budget for preparation and persistence

A duration estimate should include environment setup and model loading, not just the compute kernel. Downloading several gigabytes can take longer than a small generation task.

A useful planning decomposition is:

expected task cost =
  billed compute(setup + download + execution + output collection)
  + applicable storage
  + applicable transfer

Each term needs a source and a timestamp. If transfer charges or storage retention are unknown, report that uncertainty instead of silently assigning zero. Record the successful attempt separately from the total cost of development retries.

3. Make approval a specific contract

An account connection grants access; it is not informed approval of every future action. Before allocation, show the selected offer, minimum charge, billing increment, estimated total, hard limits, data to upload and intended output location.

Bind that approval to a current quote. If an expired quote is replaced by a different source with a different minimum charge, obtain approval for the new terms. A GPU's model name does not imply a billing policy.

4. Reconcile uncertain allocation instead of retrying blindly

A timeout is not proof that allocation failed. Creating another instance can leave two billable resources.

Persist the request identity and inspect the existing resource state before retrying. Use documented idempotency semantics where supported; do not assume a create endpoint is idempotent because another endpoint accepts an idempotency key. If the result remains uncertain, surface that state for reconciliation rather than spawning a replacement.

5. Treat output verification and release as part of success

A process exit code of zero is insufficient when the disk will disappear on termination. A task acceptance record should capture:

Workload and dependency revisions.
Execution status and exit code.
Output manifest, byte counts and checksums.
Finalized compute fees and separate storage or transfer charges.
Resource release confirmation, including whether that confirmation came from the upstream supplier.

It is useful to expose a read-only discovery endpoint for agents that have no credentials yet. Return capabilities, billing rules, task limitations, known issues and sample sizes. If there is too little production data for a success rate, return null with an explanation. A failed discovery request should remain a failure, not be replaced with a cached green check.

The practical result is a better planning boundary: the agent can explain whether a workload fits before the human grants spending authority, and the final receipt describes an actual completed task rather than just an hourly price.

Disclosure: this article was prepared with AI assistance and checked against implemented billing and task-control behavior. The examples are illustrative and do not invoke a GPU or any paid service.

Five short prompts for an agent that needs a GPU

CloudGPU — Sun, 15 Nov 2026 11:59:09 +0000

Five short prompts for an agent that needs a GPU

I work on CloudGPU. If your assistant can call GPU tools, the useful request is one short sentence: what to do, what the cap is, and when to wait. Here are five prompts you can paste, plus the short steps behind each one.

1. Rent a GPU

Prompt:

Use CloudGPU for this task: check my running instances, show the current quote and a $2 budget cap, and wait for my approval before creating anything.

Steps:

Use an MCP-capable client with CloudGPU connected (remote endpoint https://cloudgpu.app/mcp, or the local MCP server). Approving account access is not approval of this paid task.
Let the agent inspect running instances and whether this task needs a GPU.
Review the live quote, the $2 budget cap (a cap, not a GPU price), the time limit, the command, and the expected outputs. Treat storage charges as separate.
Approve only that quote, then let the agent create the rental or submit the Job. Keep the instance or Job id; do not retry an uncertain request blindly.

Guide: https://cloudgpu.app/guides/can-chatgpt-rent-and-reuse-a-cloud-gpu?utm_source=devto&utm_medium=article&utm_campaign=202611_agent_answers&utm_content=dt1115_rent

2. Reuse a running instance

Prompt:

Check whether I already have a suitable running CloudGPU instance for this task, and if it still fits, reuse it for these approved commands instead of starting a new GPU.

Steps:

Have the agent list running instances and check ownership, readiness, remaining spend, time and idle limits, and available memory or disk.
If a running rental fits, run only the newly approved commands on that same instance. Do not extend the original limits, and do not keep an idle GPU just to use remaining prepaid time.
A saved Workspace version reuses selected project files, not the operating system or installed packages. A later Job still needs a fresh quote and approval.
Separate Jobs each start and release their own GPU; they do not borrow a live rental. If applying files to an existing user-owned instance fails, do not stop that instance.

Guide: https://cloudgpu.app/guides/how-to-reuse-a-running-gpu-over-remote-mcp?utm_source=devto&utm_medium=article&utm_campaign=202611_agent_answers&utm_content=dt1115_reuse

3. Control the budget

Prompt:

For this CloudGPU task, set a $2 budget cap and a 1-hour time cap, show the current quote, and do not start until I approve.

Steps:

The $2 and 1-hour figures bound this request only. They are this-task caps, not a GPU price, and not approval for later tasks.
Review the live quote, including the billing increment, minimum charge, and time for setup, downloads, and bringing results back.
A top-up or a funded balance is not approval of this quote. Storage charges need a separate ok; they are not covered by the GPU spend cap.
After the task, check the recorded cost and confirm that a GPU rented for this task was released.

Guide: https://cloudgpu.app/guides/how-to-set-gpu-budget-caps-over-remote-mcp?utm_source=devto&utm_medium=article&utm_campaign=202611_agent_answers&utm_content=dt1115_budget

4. Run ComfyUI

Prompt:

Use CloudGPU to run this ComfyUI workflow: prefer a running instance if it already fits, keep a $2 budget cap, and download the images when it finishes.

Steps:

Prefer the official ComfyUI template. First check for a suitable running instance.
If a new rental is needed, review the live quote and this-task caps, then approve before creating it. The $2 figure is a budget cap, not a GPU price.
The template provides ComfyUI, not model weights. Download the weights the workflow needs onto the instance, into the matching folders under ComfyUI/models/.
If a public HTTPS API address is shown, it is only live while the instance runs. Submit an API-format workflow with POST /prompt, poll GET /history/{prompt_id}, and fetch images with GET /view.
Confirm the downloaded files first. Stop a rental only if this task created it. Do not stop an existing instance you were only reusing unless the owner asked.

Guide: https://cloudgpu.app/rent-gpu-for-comfyui?utm_source=devto&utm_medium=article&utm_campaign=202611_agent_answers&utm_content=dt1115_comfy

5. Save the results

Prompt:

Download this CloudGPU task’s results to ./outputs, confirm the files are there, and only then stop a GPU rented for this task.

Steps:

Stopping a rental deletes the instance disk. Saving files and stopping billing are separate steps.
Copy the results off the running machine first (a client download, rsync, or explicitly requested Job collection) and open the saved files to confirm they arrived.
If you also want a Workspace version, save selected project files after they are on the client computer. Workspace is not a full-disk backup and does not keep installed packages.
GPU spend caps do not approve extra paid storage. After the files are confirmed, stop only a GPU rented for this task. Do not destroy an existing user-owned instance you were only reusing unless the owner asked.

Guide: https://cloudgpu.app/guides/how-to-save-gpu-results-over-remote-mcp?utm_source=devto&utm_medium=article&utm_campaign=202611_agent_answers&utm_content=dt1115_save

Paste one prompt per task. The agent should wait when the quote is new, reuse a running machine when it already fits, and clean up only a rental it created for this request, after the downloads are confirmed.

How do I stop a streaming chat client from crashing on an empty delta?

CloudGPU — Wed, 11 Nov 2026 07:52:48 +0000

I work on CloudGPU. Treat streamed chunks as events, not complete chat messages. A chunk can contain a role update, no text, or no choices. Check before indexing, append only text, and keep the completion's finish reason separate from the accumulated answer.

1. Isolate the reducer from network code

Save this helper as stream_text.py. It consumes the shape of an SDK chunk after model_dump() and only follows choice index zero:

def collect_text(chunks):
    pieces = []
    finish = None
    for chunk in chunks:
        for choice in chunk.get("choices") or []:
            if choice.get("index") != 0:
                continue
            delta = choice.get("delta") or {}
            content = delta.get("content")
            if isinstance(content, str):
                pieces.append(content)
            reason = choice.get("finish_reason")
            if reason is not None:
                finish = reason
    return "".join(pieces), finish

The index is explicit: do not concatenate different alternatives into one answer. This helper handles text only. It does not interpret extra model-specific fields or claim they have the same meaning as answer text.

2. Exercise the empty cases without calling a model

In a separate local file, use a deliberately synthetic sequence:

from stream_text import collect_text

fixture = [
    {"choices": []},
    {"choices": [{"index": 0, "delta": {"role": "assistant"}}]},
    {"choices": [{"index": 0, "delta": {"content": "hello"}}]},
    {"choices": [{"index": 0, "delta": {}, "finish_reason": "stop"}]},
]
assert collect_text(fixture) == ("hello", "stop")

These events are a client-side fixture, not a captured CloudGPU response. A reducer can be tested without credentials and without inferring which optional events a particular model sends.

3. Attach the reducer to an authorized live stream

Set ${CLOUDGPU_API_KEY} and ${CLOUDGPU_BASE_URL} in the process environment. Use the documented current API base including its version prefix. Add this to a separate script when you are ready for a live request:

import os
from openai import OpenAI
from stream_text import collect_text

with OpenAI(
    api_key=os.environ["CLOUDGPU_API_KEY"],
    base_url=os.environ["CLOUDGPU_BASE_URL"],
    max_retries=0,
) as client:
    stream = client.chat.completions.create(
        model="deepseek-v4-flash",
        messages=[{"role": "user", "content": "Name one Python collection."}],
        stream=True,
    )
    try:
        text, finish = collect_text(chunk.model_dump() for chunk in stream)
    finally:
        stream.close()
    print(text)
    print({"finish_reason": finish})

An interrupted iterator raises; do not turn a partial buffer into a successful answer. Even when the iterator ends normally, a missing finish reason should remain unresolved in your application. If the reason indicates truncation, retain that state instead of labeling the answer complete. Never infer success from a nonempty text buffer alone.

Official references: Python SDK and CloudGPU's streaming contract. The model identifier was checked against today's public catalog; that is not a live-stream validation.

Related CloudGPU video: This Video Was Made by AI. Now Run AI Yourself. It provides product context, not evidence that this particular streaming sequence was observed.

The preparation checks cover syntax and an offline reducer fixture only. No API generation or GPU run was performed.

Where should gradient clipping go in a mixed-precision LoRA training step?

CloudGPU — Wed, 11 Nov 2026 06:57:33 +0000

I work on CloudGPU. When you use PyTorch gradient scaling, clip after scaler.unscale_(optimizer) and before scaler.step(optimizer). Clipping scaled gradients changes the meaning of the threshold. During gradient accumulation, wait until the effective batch is complete before unscaling.

1. Identify who owns the training step

The function below is for a custom, single-optimizer PyTorch loop using CUDA autocast with float16. It is not an instruction to add a second scaler inside a trainer that already manages mixed precision. In a managed trainer, use its documented clipping hook instead.

Construct the optimizer from the adapter parameters you actually intend to train. Frozen parameters do not become trainable just because you include them in a clipping call.

2. Put the operations in an explicit order

import torch


def adapter_step(model, optimizer, scaler, inputs, targets, loss_fn):
    # Inputs, targets and the prepared model already belong to CUDA.
    optimizer.zero_grad(set_to_none=True)
    trainable = [p for p in model.parameters() if p.requires_grad]
    if not trainable:
        raise ValueError("No trainable parameters")

    with torch.autocast(device_type="cuda", dtype=torch.float16):
        loss = loss_fn(model(inputs), targets)
    if loss.ndim != 0 or not torch.isfinite(loss.detach()).item():
        raise ValueError("Expected a finite scalar loss")

    scaler.scale(loss).backward()
    scaler.unscale_(optimizer)
    torch.nn.utils.clip_grad_norm_(
        trainable, max_norm=1.0, error_if_nonfinite=True,
    )
    scaler.step(optimizer)
    scaler.update()
    return loss.detach()

Create one torch.amp.GradScaler("cuda") for the training loop, outside this function, and pass it in. The clipping threshold is an illustrative setting, not a recommendation validated for your adapter. The function assumes one update per call and that the optimizer owns the intended trainable parameters.

The non-finite checks are a fail-fast diagnostic choice. A failed gradient check raises before the update; stop and investigate rather than catching it and silently continuing the same step. PyTorch's normal scaler-managed path can instead skip an update when gradients are non-finite. Choose the behavior deliberately.

3. Adjust the boundary for accumulation

If several microbatches form one update, zero once at the beginning of the accumulation group. Keep the scale unchanged while accumulating. Unscale and clip once after the final backward call, then step and update once. Weight each microbatch's loss according to the effective-batch reduction, including a shorter final group; do not paste this one-batch function unchanged into an accumulation loop.

Official PyTorch mixed-precision examples document unscaling, clipping and accumulation. Use your model's own forward/loss contract; the example assumes a tensor-returning model and a scalar loss.

For server-side task limits around a custom training job, see the CloudGPU integration guide.

Related CloudGPU video: This Video Was Made by AI. Now Run AI Yourself. It is remote-compute context, not a LoRA training demonstration.

This function was syntax-checked, not run as a training experiment. No convergence, memory or performance result is asserted.

Why does my PyTorch DataLoader fail when a worker touches CUDA?

CloudGPU — Wed, 11 Nov 2026 05:25:18 +0000

I work on CloudGPU. Keep dataset decoding and collation on the CPU, then move each batch to CUDA in the main training process. If a worker initializes CUDA after a fork, adding more workers can expose a process-initialization problem rather than a model problem.

1. Find the CUDA operation inside the worker

Inspect Dataset.__getitem__, transforms, collate_fn and worker_init_fn. Remove .cuda(), .to("cuda") and accelerator model construction from those paths. Also check whether a tensor was created on CUDA when the dataset object was built.

Use num_workers=0 temporarily to obtain a clearer traceback. It is a diagnostic step, not a recommendation about throughput.

2. Give the loader CPU data and an explicit process context

Save this as loader_boundary.py in an environment with a suitable PyTorch CUDA build:

import torch
from torch.utils.data import DataLoader, Dataset


class Samples(Dataset):
    def __len__(self):
        return 12

    def __getitem__(self, index):
        return torch.tensor([index], dtype=torch.float32)


def main():
    if not torch.cuda.is_available():
        raise RuntimeError("CUDA is not available in this environment")
    loader = DataLoader(
        Samples(), batch_size=3, num_workers=2,
        multiprocessing_context="spawn", pin_memory=True,
    )
    for cpu_batch in loader:
        assert cpu_batch.device.type == "cpu"
        gpu_batch = cpu_batch.to("cuda", non_blocking=True)
        assert gpu_batch.device.type == "cuda"


if __name__ == "__main__":
    main()

python3 loader_boundary.py

Keep the dataset class at module scope and the execution behind the main guard. Run the example as a script, rather than assuming an interactive notebook can spawn the same definitions. CPU-only workers still need review for picklable objects and import-time side effects.

3. Restore your real data pipeline one part at a time

Add the real decoder first, then transforms, then collation. Keep the device transfer in the main loop. This isolates the first change that reintroduces CUDA into a worker. This answer does not diagnose every worker crash: unreadable inputs and CPU resource exhaustion need separate investigation.

The official DataLoader documentation recommends CPU batches and memory pinning instead of returning CUDA tensors from multiprocess loading. The multiprocessing guidance explains accelerator initialization and process start methods.

For rental-side execution boundaries, see the CloudGPU integration guide.

Related CloudGPU video: This Video Was Made by AI. Now Run AI Yourself. It provides remote-compute context, not a demonstration or validation of this loader example.

This is a documentation-based diagnostic recipe. The code was syntax-checked locally, not executed on a GPU; no training or performance result is claimed.

Finish the GPU task with a result receipt, not just an exit code

CloudGPU — Wed, 11 Nov 2026 04:25:44 +0000

An image pipeline can finish successfully and still leave its user with the wrong deliverable. The output may be on a temporary machine, the filename may refer to an older run, or the file may exist without enough context to reproduce it.

I work on CloudGPU. This workflow checklist applies to a single AI short-drama shot as well as an adapter-training experiment.

Define delivery before execution

Before a run, write down what counts as a usable result. For a generated shot, that might mean an image sequence or a video file, a known frame size, an expected naming convention and a manifest linking it to the approved workflow. For training, it might mean an adapter, its configuration and a small evaluation set.

The acceptance rule belongs to the task proposal. “The command returned success” is a useful execution observation, but it does not establish that the deliverable can be opened or traced to the intended inputs.

Keep a receipt with the result

A practical receipt records the workflow revision, model repository and immutable revision, selected files, input checksums, seed where applicable, command arguments, environment image and output checksums. It also records the job identifier and the destination where the exported result was verified.

Those fields answer different questions. A workflow revision explains the graph. A model revision explains which weights were requested. A checksum identifies the actual transferred file. A seed helps describe the generation request, but does not by itself make execution reproducible across different software or hardware.

Keep secrets, temporary download credentials and private customer material out of the receipt. A useful public example can describe file structure and verification without publishing the files themselves.

Distinguish execution success from delivery success

Treat result delivery as a separate check:

The command reached a terminal state.
The expected output names exist.
The files are nonempty and can be decoded by the intended reader.
Export completed to the owner-approved destination.
The destination copy matches the recorded checksums.
The receipt points to that destination copy, not a temporary working path.

If export fails, the task has not delivered the result even if the generation command succeeded. Preserve a clear failure report rather than marking the handoff complete.

Choose the collection path before the job

CloudGPU documents a public compute catalog, task quotes and bounded Jobs. A Job runs its command and then destroys its GPU. Its logs are a limited execution record, not a replacement for full output files. Arrange output export inside the command before it exits, using an owner-controlled destination and the approved data-handling policy.

For an interactive instance, download and verify results before destroying it. Do not assume that stopping a local client preserves unexported files. Optional persistent files and Workspace features need a separate capabilities and account-eligibility check; they do not automatically collect a Job’s outputs or restore an entire runtime.

The current public contract separates these capabilities; check the enabled features and account eligibility before choosing a collection path.

Make another run deliberate

Once the result receipt exists, rerun decisions become easier to explain. Keep the inputs and acceptance criteria stable when testing a workflow change. If a model file, graph or environment changes, record a new revision rather than silently attaching the old receipt.

A network retry is different from a new experiment. Preserve the original job identity when reconciling an uncertain request. A deliberate rerun needs a fresh proposal, current quote, appropriate authorization and its own result receipt.

A small handoff that helps the next person

Deliver the result and its receipt together. The next person should be able to tell what was requested, what actually ran, where the output was verified and what would change in a new run. That is more useful than a folder named “final” and a screenshot of a successful command.

CloudGPU’s agent workflow guide: https://cloudgpu.app/docs/agents?utm_source=devto&utm_medium=article&utm_campaign=202611_seg01_comfyui_gpu&utm_content=article1111_result_receipt

Build and Verify Chunked Cosine Search with Python and PyTorch on a Rented GPU

CloudGPU — Wed, 11 Nov 2026 04:25:11 +0000

A vector search function can return plausible document IDs while still being wrong: a chunk offset is missing, scores are sorted along the wrong axis, or a CPU fallback hides a broken GPU environment. Before connecting search to an application, give it a fixture with answers you can check.

We will use Python and PyTorch to find five cosine neighbors for each query, stream the corpus through a rented GPU in blocks, and validate the answers against a separate CPU calculation. The complete example creates its own numerical data, so there is no model download, external dataset, or service credential to reproduce.

I work on CloudGPU.

1. Define a small, checkable retrieval task

The fixture contains 8,192 vectors, each with 128 components. We normalize every vector to unit length. For normalized vectors, cosine similarity is their dot product; larger values mean a closer direction.

The 32 queries are copies of known corpus rows. Their nearest neighbor should therefore be the copied row itself. The remaining neighbors still require an actual search. This combines a planted-answer test with a broader numerical comparison.

These are synthetic vectors, not language embeddings. They test indexing and arithmetic, not whether a search system understands a sentence. The same search function can accept your own normalized embeddings later.

The GPU part uses a block matrix multiplication and merges local top-k candidates. It does not build a document-by-document matrix. The CPU validator deliberately uses a complete query-by-corpus matrix for this small fixture, making it an independent reference for the block-merging logic.

2. Prepare a Python environment on the GPU machine

Use a Linux GPU environment with Python 3.11 and a driver compatible with the CUDA 12.8 PyTorch wheel. The CloudGPU public catalog lists RTX 4090 and RTX 5090 cards; the code checks the actual device rather than assuming the selected environment works. An existing compatible GPU environment is also suitable.

Run the following commands in a terminal on that machine:

python3.11 -m venv .venv

Save this as requirements.txt:

--index-url https://download.pytorch.org/whl/cu128
torch==2.9.1

Then install and record the resolved environment:

.venv/bin/python -m pip install -r requirements.txt
.venv/bin/python -m pip freeze > environment.txt

This selects the CUDA 12.8 wheel and PyTorch version listed on the official previous versions page. Only torch is needed; the example does not use image or audio packages. environment.txt records transitive dependencies as well as the direct requirement.

3. Save the complete search and validation script

Save the following as search.py:

import argparse
import json
import os
import platform
from pathlib import Path

# Set before importing torch or creating a CUDA context.
os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":4096:8"

import torch
import torch.nn.functional as F

SEED = 20261111
N_ROWS = 8192
N_QUERIES = 32
DIM = 128
K = 5


def fixture():
    generator = torch.Generator(device="cpu").manual_seed(SEED)
    raw = torch.randn(N_ROWS, DIM, generator=generator)
    corpus = F.normalize(raw, p=2, dim=1)
    expected_ids = torch.arange(N_QUERIES) * 137
    queries = corpus[expected_ids].clone()
    return corpus, queries, expected_ids


@torch.inference_mode()
def search(corpus, queries, device, chunk_rows):
    q = queries.to(device)
    scores = torch.empty((len(q), 0), device=device)
    ids = torch.empty((len(q), 0), dtype=torch.long, device=device)
    for offset in range(0, len(corpus), chunk_rows):
        block = corpus[offset:offset + chunk_rows].to(device)
        similarities = q @ block.T
        local_scores, local_ids = similarities.topk(
            min(K, len(block)), dim=1, sorted=True
        )
        local_ids = local_ids + offset
        candidates = torch.cat((scores, local_scores), dim=1)
        candidate_ids = torch.cat((ids, local_ids), dim=1)
        scores, positions = candidates.topk(
            min(K, candidates.shape[1]), dim=1, sorted=True
        )
        ids = candidate_ids.gather(1, positions)
        del block, similarities
    return scores.cpu(), ids.cpu()


@torch.inference_mode()
def validate(corpus, queries, expected_ids, scores, ids):
    assert scores.shape == ids.shape == (N_QUERIES, K)
    assert torch.isfinite(scores).all().item()
    assert ((ids >= 0) & (ids < len(corpus))).all().item()
    assert (scores[:, :-1] >= scores[:, 1:]).all().item()
    assert all(row.unique().numel() == K for row in ids)
    assert torch.equal(ids[:, 0], expected_ids)

    # A full matrix is acceptable only for this small CPU oracle.
    reference = queries.double() @ corpus.double().T
    expected_scores = reference.topk(K, dim=1, sorted=True).values
    selected_scores = reference.gather(1, ids)
    actual = scores.double()
    torch.testing.assert_close(
        actual, expected_scores, rtol=1e-4, atol=1e-6
    )
    torch.testing.assert_close(
        actual, selected_scores, rtol=1e-4, atol=1e-6
    )
    return (actual - expected_scores).abs().max().item()


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--device", choices=("cuda", "cpu"), default="cuda")
    parser.add_argument("--chunk-rows", type=int, default=1024)
    parser.add_argument("--output", type=Path, default=Path("search-report.json"))
    args = parser.parse_args()
    if args.chunk_rows < 1:
        parser.error("--chunk-rows must be positive")
    if args.device == "cuda" and not torch.cuda.is_available():
        raise SystemExit("CUDA is unavailable; check the driver and torch build")

    torch.manual_seed(SEED)
    torch.use_deterministic_algorithms(True)
    torch.set_float32_matmul_precision("highest")
    device = torch.device(args.device)
    corpus, queries, expected_ids = fixture()
    scores, ids = search(corpus, queries, device, args.chunk_rows)
    error = validate(corpus, queries, expected_ids, scores, ids)
    report = {
        "assertions_passed": True,
        "seed": SEED,
        "python": platform.python_version(),
        "torch": torch.__version__,
        "cuda_runtime": torch.version.cuda,
        "device": str(device),
        "device_name": torch.cuda.get_device_name(0) if device.type == "cuda" else "cpu",
        "corpus_rows": N_ROWS,
        "query_rows": N_QUERIES,
        "dimensions": DIM,
        "k": K,
        "chunk_rows": args.chunk_rows,
        "max_score_error": error,
        "expected_first_ids": expected_ids.tolist(),
        "ids": ids.tolist(),
        "scores": scores.tolist(),
    }
    args.output.write_text(
        json.dumps(report, indent=2, allow_nan=False) + "\n", encoding="utf-8"
    )
    print(f"All correctness assertions passed; wrote {args.output}")


if __name__ == "__main__":
    main()

Three details are easy to miss:

Row offsets: topk returns indices within a block. Adding offset turns them into corpus IDs. Omitting this can produce believable scores paired with incorrect documents.
The final block: min(K, len(block)) handles blocks smaller than the requested number of neighbors. The merge also accepts fewer than five candidates during early iterations.
No silent CPU fallback: the default device is CUDA. torch.cuda.is_available must succeed before the search begins. CPU mode is an explicit diagnostic choice, not a substitute reported as GPU execution.

The normalize documentation explains the row normalization. Here it runs on the CPU before transferring the same input values to the GPU, keeping data generation separate from device-specific matrix multiplication.

Why can we discard everything except a block's top five? A discarded item already has at least five items in its own block scoring no lower than it. It cannot be required to construct a global top five by score. Repeatedly taking the top five of the accumulated candidates preserves that invariant. Tied items need separate attention, discussed below.

4. Run it and inspect the evidence

Check syntax, then run the CUDA path twice:

.venv/bin/python -m py_compile search.py
.venv/bin/python search.py --device cuda --output report-a.json
.venv/bin/python search.py --device cuda --output report-b.json
.venv/bin/python -m json.tool report-a.json

Each invocation writes its report only after every assertion succeeds. The checks establish that:

every query receives five distinct, valid corpus IDs;
scores are finite and sorted in descending order;
the first match is the known source row for every query;
returned scores agree with the float64 CPU top-k scores within explicit tolerances;
each returned ID actually corresponds to its claimed score in the CPU reference.

That last check catches a subtle failure: returning the correct scores alongside unrelated IDs. Comparing scores alone would miss it. The checks use torch.testing.assert_close with both relative and absolute tolerances, and compare float64 tensors on the same device.

The report includes Python, PyTorch, CUDA runtime, device name, seed, chunk configuration, IDs, and scores. max_score_error is calculated during your execution; it is not a published measurement. This tutorial presents correctness checks, not measured GPU performance or a benchmark transcript.

For a replay check, save this as check_replay.py and run it with .venv/bin/python check_replay.py:

import json
import math
from pathlib import Path

first = json.loads(Path("report-a.json").read_text())
second = json.loads(Path("report-b.json").read_text())
assert first["assertions_passed"] and second["assertions_passed"]
for field in ("seed", "corpus_rows", "query_rows", "dimensions", "k"):
    assert first[field] == second[field]
for report in (first, second):
    assert len(report["ids"]) == report["query_rows"]
    assert [row[0] for row in report["ids"]] == report["expected_first_ids"]
for row_a, row_b in zip(first["scores"], second["scores"], strict=True):
    for a, b in zip(row_a, row_b, strict=True):
        assert math.isclose(a, b, rel_tol=1e-4, abs_tol=1e-6)
print("Replay checks passed")

Do not run the scripts with Python's optimization flag: these tutorial checks use assert statements.

You can also exercise the tail-block path:

.venv/bin/python search.py --device cuda --chunk-rows 1000 --output report-tail.json

With this configuration, the corpus does not divide evenly into blocks. A diagnostic run using --chunk-rows 3 additionally exercises blocks smaller than K; it is an edge-case test, not a suggested operating configuration.

5. Understand what reproducibility means here

A fixed CPU generator seed makes the fixture repeatable within a matching environment. The script also enables deterministic algorithms and sets the documented matrix-multiplication workspace configuration before importing PyTorch. Unsupported deterministic operations fail rather than silently switching policy.

The float32 matrix multiplication precision setting is highest, keeping the arithmetic policy explicit. None of this promises identical floating-point bytes across different hardware or PyTorch releases. PyTorch's reproducibility notes explicitly describe that boundary. Preserve the environment record, script, and reports together.

There is another boundary: torch.topk can return different indices for tied scores. The validator therefore compares top-k score values and checks the selected IDs against those values instead of requiring every secondary ID to equal the CPU's choice. The planted first-match assertion applies to this synthetic fixture, not arbitrary duplicate vectors.

If production data requires a fixed ordering for exact ties, define a secondary document-ID ordering and implement it consistently in both block selection and merging. Do not perturb the scores with arbitrary offsets and assume the ranking is unchanged.

6. Move from the fixture to application data

For your own retrieval task, replace fixture() with corpus and query vectors from the same encoder and preprocessing pipeline. Reject non-finite inputs and zero-length vectors before normalization, preserve a separate mapping from row numbers to document IDs, and normalize along the component dimension. Cosine scores from unrelated vector spaces are not meaningful.

Keep the planted fixture as a regression test. The supplied validate() is fixture-specific: for application data, remove the planted-row assertion and fixed shape constants, and run a CPU reference on a representative query sample. Do not allocate the full reference matrix for an unrestricted corpus.

Streaming corpus blocks bounds the temporary score matrix, but the example still keeps the corpus in host memory and all queries on the selected device. For a larger query set, add an outer query-batching loop. If you get an allocation error, reduce --chunk-rows first; do not infer that the arithmetic or ID mapping is correct merely because the program completes.

If CUDA is unavailable, inspect the installed PyTorch build and the environment's GPU access before proceeding. --device cpu can isolate a logic problem, but a successful CPU run does not verify the GPU path.

Make a retry answer one question

CloudGPU — Tue, 10 Nov 2026 06:11:06 +0000

I work on CloudGPU. A failed AI video shot is only useful if the next trial can tell you something new. Changing the reference, prompt and motion settings together makes that harder. Start with one question: what exactly should this retry establish?

Write an experiment card containing the observed defect, the variable to change, the acceptance rule and the output name. If the character is correct but the movement is wrong, preserve the accepted image and compare the movement stage. Keep failed outputs so a later reviewer can see why an option was rejected.

Record the workflow revision and environment alongside the seed. A seed is a comparison aid, not a promise that different software and hardware produce identical files. Check decode integrity independently from visual acceptance.

Before expanding a remote run, approve the new batch scope and its spending and time ceilings. A successful small experiment should answer a question before it becomes a larger batch.

Workspace is not a cloud desktop: four things it does not restore

CloudGPU — Mon, 09 Nov 2026 07:06:01 +0000

I work on CloudGPU, so read this as a note from the people who build it.

CloudGPU recently added Workspace: you can save project files independently of a GPU, pin them as an immutable version, and restore that version into a later Job or onto a GPU you are already renting. The most common misreading we see is "so it is a cloud desktop now". It is not, and designing around that difference saves real time.

Here are the four things Workspace does not bring back, and what to do about each.

1. Your operating system, drivers and installed packages

A Workspace version is a manifest: relative project paths mapped to files you uploaded and completed. When a Job starts with workspace_version_id, those files are restored into a fresh project directory before your command runs. That is all.

The machine underneath is whatever template the new Job or instance uses. pip install results, a compiled extension, a CUDA build you fixed by hand last week: none of that is in the version.

What to do: keep environment setup as code inside the project. A setup.sh or a pinned requirements.txt / environment.yml that your Job command runs first is the version-controlled replacement for "the machine I had".

# Job command: rebuild the environment, then run
bash setup.sh && python render.py --config configs/shot_014.yaml

2. Changes you made after saving

A version is immutable. If you edit configs/shot_014.yaml on a running GPU, the saved version still holds the old file. Nothing syncs back automatically.

What to do: treat "save a new version" as an explicit step, the same way you treat git commit. Name it after what changed (shot014-fix-lighting), not after the date.

3. Job outputs

The platform does not collect outputs for you. A Job runs one command, and the local storage of that Job's GPU is erased when it ends. If the command did not export its results, they are gone, even if the Job "succeeded".

What to do: make export part of the command, and check it as an acceptance step:

python render.py --out out/ \
  && ls out/*.png | wc -l \
  && <your upload step for out/>

Then verify on your side that the expected number of files arrived and that they open. A zero exit code is not proof that usable files reached you.

4. Unlimited time and unlimited space

Saved files are subject to account eligibility, a quota and a retention policy. Completed files carry an expires_at; downloading a file does not renew it, and a version that contains an expired file becomes unusable. Storage charges are separate from the max_spend_usd cap on a GPU Job, and a budget-limited agent key cannot switch on paid storage.

What to do: read your own limits from the authenticated GET /v1/storage before you plan a long project, and keep your own copy of anything you cannot afford to lose.

So what is it good for?

Three patterns where it genuinely helps:

Repeatable runs. Pass the same workspace_version_id to two Jobs and you know both started from byte-identical project files. If the outputs differ, the cause is elsewhere (seed, environment, upstream model), which narrows the search.
Picking up on a GPU you already rent. A saved version can be applied to a running instance. It lands in a new private directory, does not overwrite your existing files, and does not install anything, run anything or restart the GPU. You decide what happens next.
Handing a project to an agent. An agent with its own scoped key can start a Job from a version without you uploading files again in every session.

A quick mental model

Layer	Survives the GPU?	Who is responsible
OS, drivers, packages	No	Your setup script
Project files you saved as a version	Yes, within quota and retention	Workspace
Edits after the save	No, until you save a new version	You
Job outputs	No, unless exported	Your Job command

A short overview

If you have never seen CloudGPU, this short video shows what it is in general terms (it is not a Workspace tutorial):

Agent and persistence API docs: https://cloudgpu.app/docs/agents?utm_source=devto&utm_medium=article&utm_campaign=202611_seg04_codingagent&utm_content=pack1109gf_devto_workspace_boundary
Check whether persistence is enabled: GET https://cloudgpu.app/v1/public/compute-capabilities

Eligibility and full terms: https://cloudgpu.app/terms?utm_source=devto&utm_medium=article&utm_campaign=202611_seg04_codingagent&utm_content=pack1109gf_devto_workspace_boundary

What does your team keep outside the GPU today: a setup script, a container image, or something else? I am curious which part breaks most often.

Write the handoff plan before you render an AI short drama

CloudGPU — Sun, 08 Nov 2026 07:32:03 +0000

I work on CloudGPU. When a team makes an AI short drama, the hard part is often not one render. It is the chain of small decisions: which shot is kept, which inputs produced it, and which settings should carry into the next attempt. My suggestion is to write down the handoff before the render sequence starts.

1. Begin with a single shot you can review

Pick one shot that stands for the scene. Note its input assets, the workflow configuration, and what “accepted” means for it. Review that one result before you scale up. Even a rejected test is useful if the review says what has to change.

2. Store decisions next to outputs

Every selected output should have a clear home in the project, with its configuration and the inputs needed to repeat it close by. Mark which version the editor is meant to use, so nobody has to guess from a folder of near-identical files.

3. Check the copy on the receiving side

Export the chosen output plus whatever the next stage needs, then open those copies where the editor will actually work. Confirm the files are readable and the chosen version is obvious. If any of that is unchecked, the handoff stays open.

4. Treat cleanup as its own decision

The CloudGPU skill review we use notes that results must be exported before an environment is terminated. That makes “receiving copy verified” a natural milestone. Close the handoff only after it passes.

This is a proposed production method, not a benchmark: no rendered test, model capability or measured saving is claimed. For a first trial, ask an editor to receive one shot package and list every missing item before expanding the workflow.

Originally published on the CloudGPU Blog (Hashnode).

Planning an AI short-drama pipeline without buying a new GPU

CloudGPU — Sat, 07 Nov 2026 06:12:51 +0000

Before buying more hardware for AI video, compare the cost of upgrading with renting for the stages that need a GPU. Our ep05 explainer discusses that decision (video link below); this article proposes a workflow, not a completed production benchmark.

If you would rather defer a hardware purchase, one possible setup is to run the light stages through an API and rent a GPU for the stages that need your own model weights.

Untested example: The snippets below were checked against public documentation, not executed. They are illustrative, not a tested end-to-end workflow; add error handling, timeouts and output checks before use. Catalog presence is not proof of successful inference or current GPU stock.

The pipeline, stage by stage

An episode of a vertical short drama (a minute or two, 15–30 shots) breaks down like this:

Stage	What runs	Where it runs well
Series bible + episode script	a large chat model	API
Shot list + image/video prompts	a fast chat model	API
Keyframes / character sheets	FLUX	API, or ComfyUI on a GPU if you use a character LoRA
Shots	Model supporting the required input	API for supported shots; custom ComfyUI workflows on a GPU sized for the workload
Voice	TTS	API
Subtitles	Whisper	API

The rule of thumb: anything stateless and prompt-driven goes to an API. Anything that needs your weights (a character LoRA, a custom ComfyUI graph) goes on a GPU you control, for as long as the render takes.

1. Script and shot list over an OpenAI-compatible API

The OpenAI Python and Node SDKs are documented for chat completions with a CloudGPU base URL and key; that does not imply every SDK feature is supported. A large model writes the episode and a fast one turns it into shots:

import os, json
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["CLOUDGPU_API_KEY"],
    base_url="https://cloudgpu.app/v1",
)

bible = open("series_bible.md").read()   # characters, tone, recurring props

episode = client.chat.completions.create(
    model="deepseek-v4-pro",
    messages=[
        {"role": "system", "content": "You write 90-second vertical drama episodes. Keep names and facts consistent with the bible."},
        {"role": "user", "content": bible + "\n\nWrite episode 4: the sister finds the letter."},
    ],
).choices[0].message.content

shots = client.chat.completions.create(
    model="deepseek-v4-flash",
    response_format={"type": "json_object"},
    messages=[
        {"role": "system", "content": 'Split the script into shots. Return {"shots":[{"id":1,"seconds":4,"image_prompt":"...","motion_prompt":"...","line":"..."}]}'},
        {"role": "user", "content": episode},
    ],
).choices[0].message.content

shots = json.loads(shots)["shots"]
print(len(shots), "shots")

Keep the bible as a file in the repo and send it with every episode. It can help consistency, but review each generated script; it is not a guarantee.

2. Quick shots through the video endpoint

For establishing shots and B-roll that don't need a locked character, a video job is enough. It's asynchronous: create, poll, download.

import time, requests

H = {"Authorization": "Bearer " + os.environ["CLOUDGPU_API_KEY"]}
BASE = "https://cloudgpu.app/v1"

job = requests.post(f"{BASE}/videos", headers=H, json={
    "model": "wan-3.0",
    "prompt": shots[0]["image_prompt"] + ". " + shots[0]["motion_prompt"],
}).json()

while True:
    status = requests.get(f"{BASE}/videos/{job['id']}", headers=H).json()["status"]
    if status in ("completed", "failed"):
        break
    time.sleep(10)

if status == "completed":
    mp4 = requests.get(f"{BASE}/videos/{job['id']}/content", headers=H)
    open("shot_01.mp4", "wb").write(mp4.content)

Optional fields (resolution, seconds, and image for models that accept images) are listed in the API docs. The current catalog lists wan-3.0 as text-to-video only; use an image-capable model for image-to-video. Finished files are downloadable for 24 hours, so pull them into your own storage.

3. Character shots on a rented GPU

When the lead has to look the same in every shot, you want your own LoRA and your own ComfyUI graph. VRAM requirements depend on the workflow, precision, resolution and offloading. CloudGPU's published Wan 2.2 test reports about 21 GB peak VRAM for its tested 14B FP8 configuration; do not treat 24 GB or 48 GB as a universal requirement.

Instead of buying that card, start one only for the render batch. The GPU API documents server-enforced auto-shutdown limits. Set and monitor appropriate limits; they do not replace copying outputs before destruction:

templates = requests.get(f"{BASE}/gpu/templates", headers=H).json()
comfy = next(t for t in templates["data"] if "comfyui" in t["name"].lower())

inst = requests.post(f"{BASE}/gpu/instances", headers=H, json={
    "template": comfy["id"],
    "gpu_model": "auto",       # or pick a card from GET /gpu/offers
    "idle_minutes": 30,        # destroy after 30 idle minutes
    "max_minutes": 240,        # hard cap for this batch
}).json()
print(inst["id"], inst["status"])

Then SSH in, pull your LoRA and workflow, queue the shots, copy the MP4s out, and destroy the instance (DELETE /gpu/instances/{id}). Billing is per minute, rounded up, with a 1-minute minimum; billing stops when you destroy the instance. Check the live deploy page for which cards are available that day.

(The response shape above is simplified; check GET /v1/gpu/templates once and adjust the field names to what it returns.)

4. Voice and subtitles

Both are OpenAI-format audio endpoints, so the same client works:

speech = client.audio.speech.create(model="kokoro-tts", voice="af_heart", input=shots[0]["line"])
speech.write_to_file("line_01.mp3")

with open("episode_04_rough.mp4", "rb") as f:
    srt = client.audio.transcriptions.create(model="whisper-v3-turbo", file=f, response_format="srt")
open("episode_04.srt", "w").write(srt)

(Voice names depend on the TTS model; list them from the docs.)

Rent or buy?

We won't quote prices here because they move. Do the math with your own numbers:

Hours per month you actually need a big card (renders + LoRA training, not editing).
The card's local price today, plus the RAM and SSD you'd also upgrade.
How long until you'd upgrade again.

If the big-card hours are a few sessions a month, renting keeps the money in your pocket and nothing sits idle. Daily rendering can change the calculation; compare hardware, electricity, utilization and cloud costs rather than assuming buying always wins.

That's also the takeaway from ep05: buy what you need, don't hoard, and for the occasional big job, rent the kitchen instead of building one.

Video: Will phones and computers cost more next year? — https://youtu.be/XSiCDC8v18Y
Per-stage GPU guide for short dramas: https://cloudgpu.app/cloud-gpu-for-ai-short-drama?utm_source=devto&utm_medium=article&utm_campaign=202611_seg03_freelancer_gpu_api&utm_content=devto_drama_pipeline_rent

Disclosure: I work on CloudGPU. The API and GPU calls above are CloudGPU's; the pipeline is a proposal, and endpoint, model and GPU capabilities differ by provider. Pay by card or USDT; see /payments for the current list of methods.

Sources checked 2026-11-07: API docs, model catalog, billing terms, published short-drama workflow and test settings.

A model as the judge: build preference pairs with an API, then train with DPO on a rented GPU

CloudGPU — Sun, 01 Nov 2026 03:10:36 +0000

Short answer: write a short list of principles and a set of questions, have an API model write two answers to each question, have it pick the answer that follows your principles better (twice, once in each order), keep only the pairs where both picks agree, and train a LoRA adapter on those chosen and rejected pairs with DPO in LLaMA-Factory on a rented GPU. Generate the pairs on your own machine; rent the GPU only for training; copy the adapter off, then destroy the instance.

Disclosure: I work on CloudGPU. The endpoint (https://cloudgpu.app/v1) and the GPU rentals below are ours. Each step ends with a "What to check" line, because details (versions, template names, model access) change.

Where the idea comes from, and what this guide does not claim

Using a model's judgments as the training signal is usually called RL from AI Feedback (RLAIF). One public paper on it, "RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback" (Lee et al., arXiv, submitted 2023-09-01: paper), describes training a reward model on preferences generated by an off-the-shelf LLM and then using reinforcement learning. This guide keeps only the "a model states a preference between two answers" idea and skips the rest: it trains directly on the pairs with Direct Preference Optimization (paper), using LLaMA-Factory.

That makes this a small, single-developer adaptation. It is our own construction, not a result from those sources. A model's pick can be wrong, and we make no claim that the adapter you train will be better than the base model. That is why the last step is a comparison you read yourself, with a rule for when to throw the round away.

Read these things first

DeepSeek's terms. The answers and the picks in this guide come from DeepSeek, and this guide covers DeepSeek only. DeepSeek's Terms of Use (Last Update: March 27, 2026) say in section 4.2(3) that you may apply the Outputs of the Services to "a wide range of use cases, including personal use, academic research, derivative product development, training other models (such as model distillation), etc., as long as such usage is legal and adhere to these Terms." Read the terms yourself. This guide does not cover any other provider, and it does not tell you what any other provider's terms allow; if you want to use another model's outputs for training, read that provider's own terms first.
Labelling. Section 3.1 of the same terms says that if you publish or disseminate outputs, you must clearly indicate that the output content is generated by artificial intelligence. The chosen and rejected answers are outputs. If you publish the pairs, a model card or examples from them, label them as AI-generated; the script below writes a DATASET_NOTE.md you can keep next to the file.
Your data. The questions you write must be ones you have the right to use. CloudGPU's acceptable use policy (section 1.8) prohibits training on datasets you do not have rights to use. Also read the terms.
The base model's license. Each base model below has its own license. Open the link, read it, and confirm for yourself that your use is allowed.

What you need

Python 3 and the openai package on your own machine: pip install openai.
A CloudGPU account with balance, and an API key (the API page has a "Get your API key" link). Keep it in an environment variable, never in code:

export CLOUDGPU_API_KEY="paste-your-key-here"

GPU instances are billed per minute, rounded up, with a 1-minute minimum, and billing stops when you destroy the instance (terms). Rates are on the pricing page. Payment is by card or USDT (payments).
A Hugging Face account and a read token if you choose Llama or Gemma (both are gated, see the table below).

What to check: python3 -c "import os; print(bool(os.environ.get('CLOUDGPU_API_KEY')))" prints True in the shell where you will run the scripts, and python3 -c "import openai" raises no error.

API calls are billed per token from the same balance as GPU rentals; the current rate for every model is in the table on the API page. This post does not repeat rates.

Step 1: write the principles and the questions (on your own machine)

Create three plain-text files in one folder.

principles.txt is a short list, in your own words, of what a better answer looks like for your use case. An example to replace with yours:

1. Answer only what the question asks.
2. If the answer depends on information you were not given, say what is missing instead of guessing.
3. Do not invent names, numbers, links or quotes.
4. Keep the answer under 150 words unless the question asks for more.

prompts.txt is the training questions, one per line. Start with ten lines to test the script, then add more; the more varied the questions are, the more the judge has to compare. heldout.txt is 20 or so questions in the same style that you will not use for training. Write them separately, and do not copy lines between the files.

What to check: wc -l prompts.txt heldout.txt shows the line counts you expect, and no line of heldout.txt appears in prompts.txt (grep -x -F -f heldout.txt prompts.txt prints nothing).

Step 2: write two answers, let the model pick, keep consistent picks

For each question the script asks deepseek-v4.1-flash for two answers at a high temperature, then asks it twice which answer follows your principles better: once with the answers in one order and once in the other. A pair is kept only if both picks choose the same answer. Judges can lean towards whichever answer comes first, or towards the longer one; the two orders and the length count the script prints are our own suggestions for spotting that, not claims from a source. The model ID comes from the live model table on the API page (checked 2026-10-27); the same table lists deepseek-v4-flash and deepseek-v4-pro. The OpenAI SDK only needs base_url and the key changed.

# make_pairs.py
import json, os, sys, time
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
from openai import OpenAI

client = OpenAI(base_url="https://cloudgpu.app/v1", api_key=os.environ["CLOUDGPU_API_KEY"])
MODEL = "deepseek-v4.1-flash"
PRINCIPLES = Path("principles.txt").read_text(encoding="utf-8").strip()

JUDGE = """You compare two answers to the same question.
Judge them ONLY against these principles:
{principles}

Question:
{question}

Answer A:
{a}

Answer B:
{b}

Reply with exactly one letter: A if Answer A follows the principles better,
B if Answer B follows them better."""


def chat(prompt, temperature):
    for attempt in range(3):
        try:
            resp = client.chat.completions.create(
                model=MODEL,
                temperature=temperature,
                messages=[{"role": "user", "content": prompt}],
            )
            return resp.choices[0].message.content.strip()
        except Exception as err:  # network error, rate limit
            print(f"attempt {attempt + 1} failed: {err}", file=sys.stderr)
            time.sleep(2 ** attempt)
    return None


def verdict(question, first, second):
    text = chat(JUDGE.format(principles=PRINCIPLES, question=question, a=first, b=second), 0)
    letter = (text or "").strip().strip(".*").upper()
    return letter if letter in ("A", "B") else None


def make_pair(question):
    one = chat(question, 1.0)
    two = chat(question, 1.0)
    if not one or not two or one == two:
        return None
    v1 = verdict(question, one, two)  # `one` is shown first
    v2 = verdict(question, two, one)  # `two` is shown first
    if v1 is None or v2 is None:
        return None
    winner1 = "one" if v1 == "A" else "two"
    winner2 = "two" if v2 == "A" else "one"
    if winner1 != winner2:  # the judge changed its mind when the order changed
        return None
    chosen, rejected = (one, two) if winner1 == "one" else (two, one)
    return {"instruction": question, "input": "", "chosen": chosen, "rejected": rejected}


if __name__ == "__main__":
    questions = [q.strip() for q in Path("prompts.txt").read_text(encoding="utf-8").splitlines() if q.strip()]
    with ThreadPoolExecutor(max_workers=4) as pool:
        results = list(pool.map(make_pair, questions))
    pairs = [p for p in results if p and max(len(p["chosen"]), len(p["rejected"])) <= 3000]
    Path("dpo_pairs.json").write_text(json.dumps(pairs, ensure_ascii=False, indent=2), encoding="utf-8")
    Path("DATASET_NOTE.md").write_text(
        "AI-generated data: the answers in dpo_pairs.json were generated, and the preferences between "
        "them chosen, with deepseek-v4.1-flash from questions written by the dataset author. "
        "Review before use.\n", encoding="utf-8")
    longer = sum(len(p["chosen"]) > len(p["rejected"]) for p in pairs)
    print("questions:", len(questions), "pairs kept:", len(pairs), "chosen is the longer answer:", longer)

Run it first on the ten-line prompts.txt: python3 make_pairs.py. Each question takes four API calls (two answers, two picks), so a long prompts.txt makes many calls; lower max_workers if you see rate-limit errors.

What to check: dpo_pairs.json and DATASET_NOTE.md exist, and the script prints how many pairs it kept and how many times the chosen answer was the longer one. If the chosen answer is the longer one in nearly every pair, the judge may be choosing length, not your principles: tighten the principles, or drop pairs where the lengths differ a lot. If it keeps very few pairs, the two orders disagree often; read why in a few cases before you change anything. A 401 means the key variable is not set in this shell.

Step 3: read the pairs yourself

The format is LLaMA-Factory's preference format: per its data preparation docs, a preference dataset needs a better response in the chosen column and a worse one in rejected. A row looks like this (illustrative, written by hand, not model output):

{
  "instruction": "Does the config file need a restart to take effect?",
  "input": "",
  "chosen": "The documentation you gave me does not say. Check the section on reloading before relying on it.",
  "rejected": "Yes, always restart the service after editing the config file."
}

Read a sample:

python3 -c "import json,random; [print(r['instruction'], '\nCHOSEN:', r['chosen'], '\nREJECTED:', r['rejected'], '\n') for r in random.sample(json.load(open('dpo_pairs.json')), 5)]"

What to check: for each sample, ask whether you agree that chosen follows your principles better than rejected. If you disagree with several, the judge prompt or the principles are the problem, not the training: fix them and run Step 2 again. Once you are happy with the ten-line run, put all your questions in prompts.txt and run it again.

Step 4: rent a GPU

On Deploy, choose a card. The CloudGPU docs say two things that matter here: every template is an image with root SSH, and for anything a template does not cover you can rent a raw GPU (Deploy â†’ Raw GPU) and install what you need over SSH. For this guide, pick a template that includes PyTorch, or choose Raw GPU and install it yourself. Pick a card with enough memory for your base model: the LLaMA-Factory README has a rough memory table (marked as estimates), and nvidia-smi shows real use during the first training steps.

Connect with the SSH details of your running instance. Then upload the pairs from your own machine (scp takes the port with a capital -P):

scp -P <ssh-port> dpo_pairs.json <ssh-user>@<ssh-host>:/workspace/

What to check: connect with ssh -p <ssh-port> <ssh-user>@<ssh-host>, then run nvidia-smi (a GPU is listed) and python3 -c "import torch; print(torch.cuda.is_available())" (prints True). If torch is missing, install PyTorch with CUDA from the PyTorch install page first.

Step 5: install LLaMA-Factory and register the preference dataset

These are the install commands from the LLaMA-Factory README (read 2026-10-27; the repository name is spelled LlamaFactory there). Don't count on a pre-installed copy being current: check, and install if needed.

cd /workspace
git clone --depth 1 https://github.com/hiyouga/LlamaFactory.git
cd LlamaFactory
pip install -e .
pip install -r requirements/metrics.txt

Copy the pairs into data/ and add an entry to data/dataset_info.json. The "ranking": true line is what tells LLaMA-Factory this is a preference dataset:

cp /workspace/dpo_pairs.json data/
python3 - <<'EOF'
import json
p = "data/dataset_info.json"
info = json.load(open(p, encoding="utf-8"))
info["judge_pairs"] = {
    "file_name": "dpo_pairs.json",
    "ranking": True,
    "columns": {"prompt": "instruction", "query": "input", "chosen": "chosen", "rejected": "rejected"},
}
json.dump(info, open(p, "w", encoding="utf-8"), ensure_ascii=False, indent=2)
EOF

What to check: llamafactory-cli help prints the command list, and grep -A 3 judge_pairs data/dataset_info.json shows your entry with "ranking": true.

Step 6: choose a base model and train with DPO

Small models are enough to see the pipeline work. Pick one row; the template name must match the model.

Base model	LLaMA-Factory `template`	License (read it yourself)	Access
Llama 3.2 3B Instruct (`meta-llama/Llama-3.2-3B-Instruct`)	`llama3`	Llama 3.2 Community License	Gated on Hugging Face: accept the license, use a token
Gemma 3 4B IT (`google/gemma-3-4b-it`)	`gemma3`	Gemma Terms of Use	Gated on Hugging Face: accept the terms, use a token
Mistral 7B Instruct v0.3 (`mistralai/Mistral-7B-Instruct-v0.3`)	`mistral`	License shown on the model card	The model card listed no gate when we looked (2026-10-27)

Template names are from the LLaMA-Factory README's model table; license and access information was read on 2026-10-27 and can change. The Llama and Gemma licenses have conditions that can apply if you distribute or make available a model built on them. Confirm the license for your use yourself before you train or share anything.

For a gated model, request access on its Hugging Face page, then give the instance a fine-grained read token for just this run (you will revoke it at the end):

export HF_TOKEN="paste-a-read-token-here"

Put this in a file named judge_dpo.yaml in the LlamaFactory folder. The keys follow the official DPO LoRA example in the repo's examples/train_lora/ folder; the model, template and dataset lines are the parts you change:

### model
model_name_or_path: mistralai/Mistral-7B-Instruct-v0.3

### method
stage: dpo
do_train: true
finetuning_type: lora
lora_rank: 8
lora_target: all
pref_beta: 0.1
pref_loss: sigmoid

### dataset
dataset: judge_pairs
template: mistral
cutoff_len: 2048
preprocessing_num_workers: 4

### output
output_dir: saves/mistral-7b/lora/judge-dpo
logging_steps: 10
save_steps: 100
plot_loss: true
overwrite_output_dir: true

### train
per_device_train_batch_size: 1
gradient_accumulation_steps: 8
learning_rate: 5.0e-6
num_train_epochs: 3.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true

Train:

llamafactory-cli train judge_dpo.yaml

Open the LLaMA-Factory examples folder and compare your YAML with the current DPO example: key names and good values change between releases. pref_loss: sigmoid is the plain DPO loss in that example.

What to check: the log prints a training loss every 10 steps; saves/mistral-7b/lora/judge-dpo contains adapter_model.safetensors and adapter_config.json. If the base model download fails with a 401 or 403, the Hugging Face access request or token is the problem. An out-of-memory error means a smaller cutoff_len, a smaller model or a card with more memory; nvidia-smi shows the real use.

Step 7: compare the base model and the adapter on questions it never saw

A quick test in the terminal (no server is started). Create two small files in the LlamaFactory folder. infer_base.yaml:

model_name_or_path: mistralai/Mistral-7B-Instruct-v0.3
template: mistral
infer_backend: huggingface

and infer_dpo.yaml:

model_name_or_path: mistralai/Mistral-7B-Instruct-v0.3
adapter_name_or_path: saves/mistral-7b/lora/judge-dpo
template: mistral
infer_backend: huggingface

Ask each one the questions in heldout.txt (copy that file to the instance first, or paste the questions):

llamafactory-cli chat infer_base.yaml
llamafactory-cli chat infer_dpo.yaml

Read both sets of answers against your own principles.txt. A lower training loss is not the test; your reading of questions the adapter never trained on is.

What to check: for each held-out question, write down which answer follows your principles better, or "can't tell". Stop rule: if you cannot say that the adapter's answers follow your principles better more often than the base model's, throw this round away. Change the principles, the questions or the judge prompt and start again from Step 2; do not publish the adapter or write that it improved anything.

Step 8: copy the adapter off, then destroy the instance

Before you destroy anything, copy out what you want to keep: destroying the instance permanently deletes its local storage. The docs have a section called "What to save before you destroy an instance"; read it in the CloudGPU docs. From your own machine:

rsync -avP -e "ssh -p <ssh-port>" \
  <ssh-user>@<ssh-host>:/workspace/LlamaFactory/saves/mistral-7b/lora/judge-dpo/ ./adapter-judge-dpo/

Check that adapter_model.safetensors is in ./adapter-judge-dpo/, revoke the Hugging Face token you pasted on the instance (if you used one), then destroy the instance from your Dashboard (the docs: "You can destroy an instance at any time"). The minutes you spent copying are billed like any other minute, and billing stops once the instance is destroyed.

What to check: the instance no longer appears as running in your Dashboard, and ./adapter-judge-dpo/ still opens on your machine.

Privacy and access, in plain terms

Step 2 sends your questions, the principles, and the candidate answers to a model API. Don't send text you would not want processed by a model service, and read the privacy policy for what the service handles.
You keep ownership of what you put on the GPU instance. CloudGPU's terms (section 5) say it does not access the contents of your instance except where needed to investigate a credible report of a policy violation, where compelled by valid legal process, or to protect the safety of users or third parties. Local instance storage is deleted when you destroy the instance.
This guide does not start a server. If you later serve your model from an instance, use your own endpoint with your own API key: the terms (section 6) say instance service URLs are not authenticated by default, so add authentication yourself before you share the URL.

Sources

All read on 2026-10-27 (UTC+8), except the RLAIF paper, read on 2026-11-01 (UTC+8).

Lee et al., "RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback" (arXiv 2309.00267, submitted 2023-09-01, read 2026-11-01 UTC+8): paper. Direct Preference Optimization: paper.
DeepSeek, Terms of Use, Last Update March 27, 2026: cdn.deepseek.com, sections 3.1 and 4.2.
LLaMA-Factory: README and data preparation docs.
CloudGPU: API page, terms (sections 4 to 6), acceptable use policy (section 1.8, last updated April 22, 2026), privacy policy, docs.
Licenses: Llama 3.2, Gemma, Mistral 7B Instruct v0.3 model card.

DEV Community: CloudGPU

The hourly GPU price is not your agent's task budget

1. Represent the billing increment explicitly

2. Budget for preparation and persistence

3. Make approval a specific contract

4. Reconcile uncertain allocation instead of retrying blindly

5. Treat output verification and release as part of success

Five short prompts for an agent that needs a GPU

Five short prompts for an agent that needs a GPU

1. Rent a GPU

2. Reuse a running instance

3. Control the budget

4. Run ComfyUI

5. Save the results

How do I stop a streaming chat client from crashing on an empty delta?

1. Isolate the reducer from network code

2. Exercise the empty cases without calling a model

3. Attach the reducer to an authorized live stream

Where should gradient clipping go in a mixed-precision LoRA training step?

1. Identify who owns the training step

2. Put the operations in an explicit order

3. Adjust the boundary for accumulation

Why does my PyTorch DataLoader fail when a worker touches CUDA?

1. Find the CUDA operation inside the worker

2. Give the loader CPU data and an explicit process context

3. Restore your real data pipeline one part at a time

Finish the GPU task with a result receipt, not just an exit code

Define delivery before execution

Keep a receipt with the result

Distinguish execution success from delivery success

Choose the collection path before the job

Make another run deliberate

A small handoff that helps the next person

Build and Verify Chunked Cosine Search with Python and PyTorch on a Rented GPU

1. Define a small, checkable retrieval task

2. Prepare a Python environment on the GPU machine

3. Save the complete search and validation script

4. Run it and inspect the evidence

5. Understand what reproducibility means here

6. Move from the fixture to application data

Related video and cleanup

Make a retry answer one question

Workspace is not a cloud desktop: four things it does not restore

1. Your operating system, drivers and installed packages

2. Changes you made after saving

3. Job outputs

4. Unlimited time and unlimited space

So what is it good for?

A quick mental model

A short overview

Read more

Write the handoff plan before you render an AI short drama

1. Begin with a single shot you can review

2. Store decisions next to outputs

3. Check the copy on the receiving side

4. Treat cleanup as its own decision

Planning an AI short-drama pipeline without buying a new GPU

The pipeline, stage by stage

1. Script and shot list over an OpenAI-compatible API

2. Quick shots through the video endpoint

3. Character shots on a rented GPU

4. Voice and subtitles

Rent or buy?

A model as the judge: build preference pairs with an API, then train with DPO on a rented GPU

Where the idea comes from, and what this guide does not claim

Read these things first

What you need

Step 1: write the principles and the questions (on your own machine)

Step 2: write two answers, let the model pick, keep consistent picks

Step 3: read the pairs yourself

Step 4: rent a GPU

Step 5: install LLaMA-Factory and register the preference dataset

Step 6: choose a base model and train with DPO

Step 7: compare the base model and the adapter on questions it never saw

Step 8: copy the adapter off, then destroy the instance

Privacy and access, in plain terms

Sources