A model as the judge: build preference pairs with an API, then train with DPO on a rented GPU

Short answer: write a short list of principles and a set of questions, have an API model write two answers to each question, have it pick the answer that follows your principles better (twice, once in each order), keep only the pairs where both picks agree, and train a LoRA adapter on those chosen and rejected pairs with DPO in LLaMA-Factory on a rented GPU. Generate the pairs on your own machine; rent the GPU only for training; copy the adapter off, then destroy the instance.

Disclosure: I work on CloudGPU. The endpoint (https://cloudgpu.app/v1) and the GPU rentals below are ours. Each step ends with a "What to check" line, because details (versions, template names, model access) change.

Where the idea comes from, and what this guide does not claim

Anthropic's Constitutional AI work (2022-12-15) has a stage in which the model samples two answers, a model evaluates which of the two is better against written principles, and the resulting AI preferences are used for training: research page, paper. The paper then trains a preference model and uses reinforcement learning. This guide keeps only the "a model compares two samples" idea and skips the rest: it trains directly on the pairs with Direct Preference Optimization (paper), using LLaMA-Factory.

That makes this a small, single-developer adaptation. It is our own construction, not a result from those sources. A model's pick can be wrong, and we make no claim that the adapter you train will be better than the base model. That is why the last step is a comparison you read yourself, with a rule for when to throw the round away.

Read these things first

1. DeepSeek's terms. The answers and the picks in this guide come from DeepSeek, and this guide covers DeepSeek only. DeepSeek's Terms of Use (Last Update: March 27, 2026) say in section 4.2(3) that you may apply the Outputs of the Services to "a wide range of use cases, including personal use, academic research, derivative product development, training other models (such as model distillation), etc., as long as such usage is legal and adhere to these Terms." Read the terms yourself. This guide does not cover any other provider, and it does not tell you what any other provider's terms allow; if you want to use another model's outputs for training, read that provider's own terms first.
2. Labelling. Section 3.1 of the same terms says that if you publish or disseminate outputs, you must clearly indicate that the output content is generated by artificial intelligence. The chosen and rejected answers are outputs. If you publish the pairs, a model card or examples from them, label them as AI-generated; the script below writes a DATASET_NOTE.md you can keep next to the file.
3. Your data. The questions you write must be ones you have the right to use. CloudGPU's acceptable use policy (section 1.8) prohibits training on datasets you do not have rights to use. Also read the terms.
4. The base model's license. Each base model below has its own license. Open the link, read it, and confirm for yourself that your use is allowed.

What you need

• Python 3 and the openai package on your own machine: pip install openai.
• A CloudGPU account with balance, and an API key (the API page has a "Get your API key" link). Keep it in an environment variable, never in code:

export CLOUDGPU_API_KEY="paste-your-key-here"

• GPU instances are billed per minute, rounded up, with a 1-minute minimum, and billing stops when you destroy the instance (terms). Rates are on the pricing page. Payment is by card or USDT (payments).
• A Hugging Face account and a read token if you choose Llama or Gemma (both are gated, see the table below).

What to check: python3 -c "import os; print(bool(os.environ.get('CLOUDGPU_API_KEY')))" prints True in the shell where you will run the scripts, and python3 -c "import openai" raises no error.

API calls are billed per token from the same balance as GPU rentals; the current rate for every model is in the table on the API page. This post does not repeat rates.

Step 1: write the principles and the questions (on your own machine)

Create three plain-text files in one folder.

principles.txt is a short list, in your own words, of what a better answer looks like for your use case. An example to replace with yours:

1. Answer only what the question asks.
2. If the answer depends on information you were not given, say what is missing instead of guessing.
3. Do not invent names, numbers, links or quotes.
4. Keep the answer under 150 words unless the question asks for more.

prompts.txt is the training questions, one per line. Start with ten lines to test the script, then add more; the more varied the questions are, the more the judge has to compare. heldout.txt is 20 or so questions in the same style that you will not use for training. Write them separately, and do not copy lines between the files.

What to check: wc -l prompts.txt heldout.txt shows the line counts you expect, and no line of heldout.txt appears in prompts.txt (grep -x -F -f heldout.txt prompts.txt prints nothing).

Step 2: write two answers, let the model pick, keep consistent picks

For each question the script asks deepseek-v4.1-flash for two answers at a high temperature, then asks it twice which answer follows your principles better: once with the answers in one order and once in the other. A pair is kept only if both picks choose the same answer. Judges can lean towards whichever answer comes first, or towards the longer one; the two orders and the length count the script prints are our own suggestions for spotting that, not claims from a source. The model ID comes from the live model table on the API page (checked 2026-10-27); the same table lists deepseek-v4-flash and deepseek-v4-pro. The OpenAI SDK only needs base_url and the key changed.

# make_pairs.py
import json, os, sys, time
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
from openai import OpenAI

client = OpenAI(base_url="https://cloudgpu.app/v1", api_key=os.environ["CLOUDGPU_API_KEY"])
MODEL = "deepseek-v4.1-flash"
PRINCIPLES = Path("principles.txt").read_text(encoding="utf-8").strip()

JUDGE = """You compare two answers to the same question.
Judge them ONLY against these principles:
{principles}

Question:
{question}

Answer A:
{a}

Answer B:
{b}

Reply with exactly one letter: A if Answer A follows the principles better,
B if Answer B follows them better."""


def chat(prompt, temperature):
    for attempt in range(3):
        try:
            resp = client.chat.completions.create(
                model=MODEL,
                temperature=temperature,
                messages=[{"role": "user", "content": prompt}],
            )
            return resp.choices[0].message.content.strip()
        except Exception as err:  # network error, rate limit
            print(f"attempt {attempt + 1} failed: {err}", file=sys.stderr)
            time.sleep(2 ** attempt)
    return None


def verdict(question, first, second):
    text = chat(JUDGE.format(principles=PRINCIPLES, question=question, a=first, b=second), 0)
    letter = (text or "").strip().strip(".*").upper()
    return letter if letter in ("A", "B") else None


def make_pair(question):
    one = chat(question, 1.0)
    two = chat(question, 1.0)
    if not one or not two or one == two:
        return None
    v1 = verdict(question, one, two)  # `one` is shown first
    v2 = verdict(question, two, one)  # `two` is shown first
    if v1 is None or v2 is None:
        return None
    winner1 = "one" if v1 == "A" else "two"
    winner2 = "two" if v2 == "A" else "one"
    if winner1 != winner2:  # the judge changed its mind when the order changed
        return None
    chosen, rejected = (one, two) if winner1 == "one" else (two, one)
    return {"instruction": question, "input": "", "chosen": chosen, "rejected": rejected}


if __name__ == "__main__":
    questions = [q.strip() for q in Path("prompts.txt").read_text(encoding="utf-8").splitlines() if q.strip()]
    with ThreadPoolExecutor(max_workers=4) as pool:
        results = list(pool.map(make_pair, questions))
    pairs = [p for p in results if p and max(len(p["chosen"]), len(p["rejected"])) <= 3000]
    Path("dpo_pairs.json").write_text(json.dumps(pairs, ensure_ascii=False, indent=2), encoding="utf-8")
    Path("DATASET_NOTE.md").write_text(
        "AI-generated data: the answers in dpo_pairs.json were generated, and the preferences between "
        "them chosen, with deepseek-v4.1-flash from questions written by the dataset author. "
        "Review before use.\n", encoding="utf-8")
    longer = sum(len(p["chosen"]) > len(p["rejected"]) for p in pairs)
    print("questions:", len(questions), "pairs kept:", len(pairs), "chosen is the longer answer:", longer)

Run it first on the ten-line prompts.txt: python3 make_pairs.py. Each question takes four API calls (two answers, two picks), so a long prompts.txt makes many calls; lower max_workers if you see rate-limit errors.

What to check: dpo_pairs.json and DATASET_NOTE.md exist, and the script prints how many pairs it kept and how many times the chosen answer was the longer one. If the chosen answer is the longer one in nearly every pair, the judge may be choosing length, not your principles: tighten the principles, or drop pairs where the lengths differ a lot. If it keeps very few pairs, the two orders disagree often; read why in a few cases before you change anything. A 401 means the key variable is not set in this shell.

Step 3: read the pairs yourself

The format is LLaMA-Factory's preference format: per its data preparation docs, a preference dataset needs a better response in the chosen column and a worse one in rejected. A row looks like this (illustrative, written by hand, not model output):

{
  "instruction": "Does the config file need a restart to take effect?",
  "input": "",
  "chosen": "The documentation you gave me does not say. Check the section on reloading before relying on it.",
  "rejected": "Yes, always restart the service after editing the config file."
}

Read a sample:

python3 -c "import json,random; [print(r['instruction'], '\nCHOSEN:', r['chosen'], '\nREJECTED:', r['rejected'], '\n') for r in random.sample(json.load(open('dpo_pairs.json')), 5)]"

What to check: for each sample, ask whether you agree that chosen follows your principles better than rejected. If you disagree with several, the judge prompt or the principles are the problem, not the training: fix them and run Step 2 again. Once you are happy with the ten-line run, put all your questions in prompts.txt and run it again.

Step 4: rent a GPU

On Deploy, choose a card. The CloudGPU docs say two things that matter here: every template is an image with root SSH, and for anything a template does not cover you can rent a raw GPU (Deploy → Raw GPU) and install what you need over SSH. For this guide, pick a template that includes PyTorch, or choose Raw GPU and install it yourself. Pick a card with enough memory for your base model: the LLaMA-Factory README has a rough memory table (marked as estimates), and nvidia-smi shows real use during the first training steps.

Connect with the SSH details of your running instance. Then upload the pairs from your own machine (scp takes the port with a capital -P):

scp -P <ssh-port> dpo_pairs.json <ssh-user>@<ssh-host>:/workspace/

What to check: connect with ssh -p <ssh-port> <ssh-user>@<ssh-host>, then run nvidia-smi (a GPU is listed) and python3 -c "import torch; print(torch.cuda.is_available())" (prints True). If torch is missing, install PyTorch with CUDA from the PyTorch install page first.

Step 5: install LLaMA-Factory and register the preference dataset

These are the install commands from the LLaMA-Factory README (read 2026-10-27; the repository name is spelled LlamaFactory there). Don't count on a pre-installed copy being current: check, and install if needed.

cd /workspace
git clone --depth 1 https://github.com/hiyouga/LlamaFactory.git
cd LlamaFactory
pip install -e .
pip install -r requirements/metrics.txt

Copy the pairs into data/ and add an entry to data/dataset_info.json. The "ranking": true line is what tells LLaMA-Factory this is a preference dataset:

cp /workspace/dpo_pairs.json data/
python3 - <<'EOF'
import json
p = "data/dataset_info.json"
info = json.load(open(p, encoding="utf-8"))
info["judge_pairs"] = {
    "file_name": "dpo_pairs.json",
    "ranking": True,
    "columns": {"prompt": "instruction", "query": "input", "chosen": "chosen", "rejected": "rejected"},
}
json.dump(info, open(p, "w", encoding="utf-8"), ensure_ascii=False, indent=2)
EOF

What to check: llamafactory-cli help prints the command list, and grep -A 3 judge_pairs data/dataset_info.json shows your entry with "ranking": true.

Step 6: choose a base model and train with DPO

Small models are enough to see the pipeline work. Pick one row; the template name must match the model.

Base model	LLaMA-Factory template	License (read it yourself)	Access
Llama 3.2 3B Instruct (meta-llama/Llama-3.2-3B-Instruct)	llama3	Llama 3.2 Community License	Gated on Hugging Face: accept the license, use a token
Gemma 3 4B IT (google/gemma-3-4b-it)	gemma3	Gemma Terms of Use	Gated on Hugging Face: accept the terms, use a token
Mistral 7B Instruct v0.3 (mistralai/Mistral-7B-Instruct-v0.3)	mistral	License shown on the model card	The model card listed no gate when we looked (2026-10-27)

Template names are from the LLaMA-Factory README's model table; license and access information was read on 2026-10-27 and can change. The Llama and Gemma licenses have conditions that can apply if you distribute or make available a model built on them. Confirm the license for your use yourself before you train or share anything.

For a gated model, request access on its Hugging Face page, then give the instance a fine-grained read token for just this run (you will revoke it at the end):

export HF_TOKEN="paste-a-read-token-here"

Put this in a file named judge_dpo.yaml in the LlamaFactory folder. The keys follow the official DPO LoRA example in the repo's examples/train_lora/ folder; the model, template and dataset lines are the parts you change:

### model
model_name_or_path: mistralai/Mistral-7B-Instruct-v0.3

### method
stage: dpo
do_train: true
finetuning_type: lora
lora_rank: 8
lora_target: all
pref_beta: 0.1
pref_loss: sigmoid

### dataset
dataset: judge_pairs
template: mistral
cutoff_len: 2048
preprocessing_num_workers: 4

### output
output_dir: saves/mistral-7b/lora/judge-dpo
logging_steps: 10
save_steps: 100
plot_loss: true
overwrite_output_dir: true

### train
per_device_train_batch_size: 1
gradient_accumulation_steps: 8
learning_rate: 5.0e-6
num_train_epochs: 3.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true

Train:

llamafactory-cli train judge_dpo.yaml

Open the LLaMA-Factory examples folder and compare your YAML with the current DPO example: key names and good values change between releases. pref_loss: sigmoid is the plain DPO loss in that example.

What to check: the log prints a training loss every 10 steps; saves/mistral-7b/lora/judge-dpo contains adapter_model.safetensors and adapter_config.json. If the base model download fails with a 401 or 403, the Hugging Face access request or token is the problem. An out-of-memory error means a smaller cutoff_len, a smaller model or a card with more memory; nvidia-smi shows the real use.

Step 7: compare the base model and the adapter on questions it never saw

A quick test in the terminal (no server is started). Create two small files in the LlamaFactory folder. infer_base.yaml:

model_name_or_path: mistralai/Mistral-7B-Instruct-v0.3
template: mistral
infer_backend: huggingface

and infer_dpo.yaml:

model_name_or_path: mistralai/Mistral-7B-Instruct-v0.3
adapter_name_or_path: saves/mistral-7b/lora/judge-dpo
template: mistral
infer_backend: huggingface

Ask each one the questions in heldout.txt (copy that file to the instance first, or paste the questions):

llamafactory-cli chat infer_base.yaml
llamafactory-cli chat infer_dpo.yaml

Read both sets of answers against your own principles.txt. A lower training loss is not the test; your reading of questions the adapter never trained on is.

What to check: for each held-out question, write down which answer follows your principles better, or "can't tell". Stop rule: if you cannot say that the adapter's answers follow your principles better more often than the base model's, throw this round away. Change the principles, the questions or the judge prompt and start again from Step 2; do not publish the adapter or write that it improved anything.

Step 8: copy the adapter off, then destroy the instance

Before you destroy anything, copy out what you want to keep: destroying the instance permanently deletes its local storage. The docs have a section called "What to save before you destroy an instance"; read it in the CloudGPU docs. From your own machine:

rsync -avP -e "ssh -p <ssh-port>" \
  <ssh-user>@<ssh-host>:/workspace/LlamaFactory/saves/mistral-7b/lora/judge-dpo/ ./adapter-judge-dpo/

Check that adapter_model.safetensors is in ./adapter-judge-dpo/, revoke the Hugging Face token you pasted on the instance (if you used one), then destroy the instance from your Dashboard (the docs: "You can destroy an instance at any time"). The minutes you spent copying are billed like any other minute, and billing stops once the instance is destroyed.

What to check: the instance no longer appears as running in your Dashboard, and ./adapter-judge-dpo/ still opens on your machine.

Privacy and access, in plain terms

• Step 2 sends your questions, the principles, and the candidate answers to a model API. Don't send text you would not want processed by a model service, and read the privacy policy for what the service handles.
• You keep ownership of what you put on the GPU instance. CloudGPU's terms (section 5) say it does not access the contents of your instance except where needed to investigate a credible report of a policy violation, where compelled by valid legal process, or to protect the safety of users or third parties. Local instance storage is deleted when you destroy the instance.
• This guide does not start a server. If you later serve your model from an instance, use your own endpoint with your own API key: the terms (section 6) say instance service URLs are not authenticated by default, so add authentication yourself before you share the URL.

Sources

All read on 2026-10-27 (UTC+8).

• Anthropic, Constitutional AI (2022-12-15): research page, paper. Direct Preference Optimization: paper.
• DeepSeek, Terms of Use, Last Update March 27, 2026: cdn.deepseek.com, sections 3.1 and 4.2.
• LLaMA-Factory: README and data preparation docs.
• CloudGPU: API page, terms (sections 4 to 6), acceptable use policy (section 1.8, last updated April 22, 2026), privacy policy, docs.
• Licenses: Llama 3.2, Gemma, Mistral 7B Instruct v0.3 model card.