DEV Community

Cover image for Self-Play Data Factory
Harish Kotra (he/him)
Harish Kotra (he/him)

Posted on

Self-Play Data Factory

#ai #programming #productivity #dailybuild2026

Turn one LLM into a training-data factory. A single model generates N diverse candidate answers, scores each one with a fine-grained A–T letter-scale logprob verifier, and emits a clean DPO preference dataset — no second model, no human labels, no binary "good vs bad".

Table of contents

Why this exists

Training-data generation for DPO/RLHF usually needs a second, stronger
model as a judge, or a human in the loop. This project shows you can get
surprisingly good preference data from one model playing both roles:

  1. Actor — generates N candidate answers to a prompt.
  2. Critic — the same model scores each candidate on a fine-grained 20-letter scale, and we read the token-logprob distribution (not just the sampled letter) to recover a calibrated 0–1 reward.

The trick that makes it work is the letter-scale logprob expectation
(replicated from the LLM-as-a-Verifier
technique). Ask a model to "rate 1–10" and you get a coarse, noisy integer.
Ask it for one letter token and read the softmax over the letters it was
torn between — you get a fine-grained reward (0.842 vs 0.860) that lets
the factory pick genuinely better candidates and emit clean preference
pairs, instead of arbitrary good-vs-bad flips.

Features

  • N candidate generation with a temperature sweep (0.3–1.2) for diversity and exact-duplicate dedup.
  • A–T letter-scale verifier (20 letters) that reads top_logprobs: 20 and computes the expectation over the letter distribution.
  • Criteria decomposition — scores each candidate against multiple criteria (Correctness, Clarity, Efficiency, …), user-defined or auto-generated from the prompt.
  • Repeated evaluation (--n-evaluations N) to average out variance.
  • DPO preference pairs — chosen = top candidate, rejected = bottom, emitted only when the score gap is meaningful (Δ > 0.05 default).
  • Zero SDK, zero framework — global fetch only. Talks to any OpenAI-compatible endpoint: DeepSeek, OpenAI, vLLM, Ollama, LM Studio.
  • Deterministic mock LLM — runs end to end with no API key, while still exercising the full logprob path.
  • Friendly CLI — a helpful hint (not a crash) when no key is present.
  • Unit-tested math — the expectation core is pinned down with node:test.

Architecture 

                         ┌─────────────────────────────┐
                         │       src/index.ts (CLI)    │
                         │  --mock --prompt --candidates│
                         │  --criteria --n-evaluations │
                         └──────────────┬──────────────┘
                                        │
                                        ▼
                         ┌─────────────────────────────┐
                         │      src/factory.ts         │
                         │  runFactory(): orchestrates │
                         └───────┬──────────┬──────────┘
                                 │          │
                 generate        │          │  verify
                                 ▼          ▼
         ┌───────────────────────┐   ┌─────────────────────────────┐
         │   src/generator.ts    │   │       src/verifier.ts       │
         │ N calls, temp sweep,  │   │ A–T letter-scale logprob    │
         │ exact dedupe          │   │ expectation scorer          │
         └───────────┬───────────┘   └──────────────┬──────────────┘
                     │                              │
                     └──────────────┬───────────────┘
                                    │
                                    ▼
                     ┌─────────────────────────────┐
                     │   src/llm.ts                │
                     │ OpenAICompatibleClient      │
                     │ chat() + chatWithLogprobs() │
                     │ (global fetch, no SDK)      │
                     └──────────────┬──────────────┘
                                    │
                    POST /v1/chat/completions
                                    │
                          ┌─────────▼─────────┐
                          │  Any OpenAI-      │
                          │  compatible       │
                          │  endpoint         │
                          │ (DeepSeek/vLLM/   │
                          │  Ollama/LM Studio)│
                          └───────────────────┘
Enter fullscreen mode Exit fullscreen mode

The data flow 

prompt
  │
  ▼ 1. generateCandidates(prompt, N, {temperature sweep})
  ▼
N candidates ──► dedupe exact duplicates
  │
  ▼ 2. resolveCriteria (explicit --criteria or auto-generated)
  │
  ▼ 3. scoreCandidates(prompt, candidates, criteria)
  │       per (candidate, criterion):
  │         chatWithLogprobs → first-token top-logprobs (A–T)
  │         expected = Σ P(letter) · score(letter)   ← fine-grained reward
  │         averaged over N evaluations
  │
  ▼ 4. rank descending by score
  │
  ├──► 5. print leaderboard (score + per-criterion + snippet)
  │
  └──► 6. buildPairs: chosen = top, rejected = bottom
            emit only when score_delta > minScoreDelta (default 0.05)
          │
          ▼ 7. write data/dataset.jsonl (DPO format)
Enter fullscreen mode Exit fullscreen mode

Technologies

Layer Choice Why
Language TypeScript (ESM, strict) Type safety across the pipeline; .ts extension imports, verbatimModuleSyntax, noUnusedLocals
Runtime Node 18+ Global fetch and AbortController — no HTTP SDK
Runner tsx Run TypeScript directly, zero build step
HTTP global fetch Talks to any OpenAI-compatible /v1/chat/completions
Config hand-rolled .env parser No dotenv dependency (splits lines on =)
Tests node:test (built-in) No test framework dependency
Dev deps typescript, tsx, @types/node The only three

Dependencies (runtime): zero. package.json has an empty
dependencies block — the entire project runs on Node's built-ins.

Project layout 

selfplay-factory/
├── package.json          # type:module; scripts: demo, demo:mock, typecheck, test
├── tsconfig.json         # strict, ESNext, moduleResolution bundler, noEmit
├── .env.example          # OPENAI_BASE_URL / OPENAI_API_KEY / OPENAI_MODEL / VERIFIER_MODEL
├── .gitignore            # node_modules, .env, data/*.jsonl, etc.
├── README.md             # this file
└── src/
    ├── config.ts         # manual .env parser → Config
    ├── llm.ts            # OpenAICompatibleClient: chat() + chatWithLogprobs()
    ├── generator.ts      # generateCandidates: N calls, temp sweep, dedupe
    ├── verifier.ts       # THE core: A–T letter-scale logprob expectation
    ├── prompts.ts        # generator + verifier-rubric + criteria-gen templates
    ├── factory.ts        # orchestration → rank → pairs → dataset.jsonl → leaderboard
    ├── mock.ts           # deterministic mock LLM with a full logprob distribution
    ├── index.ts          # CLI entry
    └── verifier.test.ts  # node:test unit tests for the expectation math
Enter fullscreen mode Exit fullscreen mode

Quickstart 

npm i

# Demo with the deterministic mock LLM — no API key needed.
npm run demo:mock
# or
npx tsx src/index.ts --mock --prompt "Write a hello world" --candidates 5 --n-evaluations 1
Enter fullscreen mode Exit fullscreen mode

That prints a leaderboard and writes data/dataset.jsonl:

{"prompt":"Write a hello world","chosen":"Here is a robust hello world: ...","rejected":"I don't know. I cannot answer this question.","score_chosen":0.842,"score_rejected":0.004,"score_delta":0.838}
Enter fullscreen mode Exit fullscreen mode

Run the tests and typecheck:

npm test          # node:test — 16 unit tests
npm run typecheck # tsc --noEmit — must pass clean
Enter fullscreen mode Exit fullscreen mode

Point it at a real model (DeepSeek v4 flash) 

cp .env.example .env
Enter fullscreen mode Exit fullscreen mode 
OPENAI_BASE_URL=https://api.deepseek.com/v1
OPENAI_API_KEY=sk-…
OPENAI_MODEL=deepseek-v4-flash-0731
VERIFIER_MODEL=deepseek-v4-flash-0731   # optional; defaults to OPENAI_MODEL
Enter fullscreen mode Exit fullscreen mode

Then:

npm run demo     # real endpoint, 5 candidates, 1 evaluation
Enter fullscreen mode Exit fullscreen mode

Any OpenAI-compatible endpoint works — DeepSeek, OpenAI, vLLM
(http://localhost:8000/v1), Ollama (http://localhost:11434/v1), LM Studio.

No key + no --mock? The CLI prints a friendly hint instead of
crashing. It also explains exactly how to point at DeepSeek v4 flash.

CLI reference 

npx tsx src/index.ts [flags]

--mock                 Use the deterministic mock LLM (no API key needed)
--prompt "…"           Prompt to generate candidates for
--candidates N         How many candidates to generate        (default 5)
--criteria "A,B,C"     Explicit criteria; omit to auto-generate
--n-evaluations N      Average N repeated scores per criterion (default 1)
--min-score-delta X    Min score gap to emit a pair             (default 0.05)
--out-file PATH        Output path                              (default data/dataset.jsonl)
-h, --help             Help
Enter fullscreen mode Exit fullscreen mode

Example:

npx tsx src/index.ts --mock --prompt "Write a fibonacci function" \
  --candidates 4 --criteria "Correctness,Clarity,Efficiency" --n-evaluations 3
Enter fullscreen mode Exit fullscreen mode

The verifier algorithm — the heart of it

Naive scoring asks the model to "rate 1–10" and trusts the single sampled
integer — a coarse, noisy, hard-to-calibrate signal. This project replicates
the LLM-as-a-verifier technique:

1. Letter scale A–T (20 letters), not digits

The verifier is told to answer with exactly one letter. Letters force a
clean single-token distribution. The prompt (src/prompts.ts):

# Reply format
Reply with exactly one letter. No punctuation, no explanation.

Available letters:
  A
  B
  ...
  T
Enter fullscreen mode Exit fullscreen mode

2. Logprob extraction

Every verification call sends logprobs: true, top_logprobs: 20 and reads
the first token position's top-logprob list (src/llm.ts):

async chatWithLogprobs(message: ChatMessage, opts: ChatOptions = {}) {
  const data = await this.post(this.buildBody(message, { ...opts, logprobs: true }));
  const firstToken = data?.choices[0]?.logprobs?.content?.[0] ?? null;
  const topLogprobs = firstToken?.top_logprobs ?? null;
  return { text: content.trim(), topLogprobs };
}
Enter fullscreen mode Exit fullscreen mode

The CLI logs what arrives, so you can watch the raw distribution:

[llm] first token="B" top=["B","A","C"]
Enter fullscreen mode Exit fullscreen mode

3. Expectation over the distribution

Each observed letter maps to a score in [0,1], evenly spaced:
A = 1.0, B = 1 − 1/19, …, T = 0.0 (scoreLetter):

export function scoreLetter(letter: string | null): number {
  const idx = LETTERS.indexOf(letter.toUpperCase());
  return SCORE_MAX - (idx / (LETTER_COUNT - 1)) * (SCORE_MAX - SCORE_MIN);
}
Enter fullscreen mode Exit fullscreen mode

The score is the expected value over the letter distribution
(expectedScoreFromLogprobs):

expected = Σ P(letter) · score(letter)
Enter fullscreen mode Exit fullscreen mode

where P(letter) is the softmax of the letter's logprob, renormalized over
the letters that actually appeared (out-of-scale tokens like U or
multi-char tokens are dropped):

export function expectedScoreFromLogprobs(topLogprobs: TokenLogprob[] | null): number | null {
  // ...collect the A–T letters that appeared with their logprobs...
  let z = 0;
  for (const s of seen) {
    s.p = Math.exp(Math.log(s.p) - logSum); // normalize by max-logprob
    z += s.p;
  }
  let expected = 0;
  for (const s of seen) {
    expected += (s.p / z) * scoreLetter(s.letter);
  }
  return expected;
}
Enter fullscreen mode Exit fullscreen mode

If the model is torn between B and A, the score lands between 0.947
and 1.0 instead of flipping a coin — that's the fine-grained signal that
makes the factory work.

4. Criteria decomposition

Each candidate is scored against multiple criteria (Correctness,
Clarity, Efficiency, …), user-provided or auto-generated from the prompt.
The final score is the mean over criteria:

const scores = details.map((d) => d.score).filter((s) => !Number.isNaN(s));
const final = scores.length > 0 ? scores.reduce((a, b) => a + b, 0) / scores.length : NaN;
Enter fullscreen mode Exit fullscreen mode

5. Repeated evaluation (optional)

--n-evaluations N runs each (candidate, criterion) pair N times and
averages, reducing per-sample variance:

const score = used.length > 0 ? used.reduce((a, b) => a + b, 0) / used.length : Number.NaN;
Enter fullscreen mode Exit fullscreen mode

Fallbacks

Some endpoints ignore logprobs. If no logprob list comes back, the verifier
parses the letter from the text; if there's no letter, it parses a plain
0–10 / 0–1 number. All of this is unit-tested.

The factory pipeline

src/factory.ts orchestrates the whole thing:

export async function runFactory(prompt: string, opts: FactoryOptions): Promise<FactoryResult> {
  // 1. Generate candidates.
  const candidates = await generateCandidates(prompt, nCandidates, { client, model, log });

  // 2. Resolve criteria (explicit or auto-generated).
  const criteria = await resolveCriteria(client, cfg, prompt, opts.criteria ?? [], log);

  // 3. Verify every candidate.
  const scored = await scoreCandidates(prompt, candidates, criteria, { client, model, nEvaluations, log });

  // 4. Rank descending.
  const ranked = [...scored].sort((a, b) => b.score - a.score);

  // 5. Build preference pairs (chosen = top, rejected = bottom).
  const pairs = buildPairs(prompt, ranked, minDelta);

  // 6. Write data/dataset.jsonl (DPO format).
  writeFileSync(resolve(outFile), lines.join('\n') + '\n', 'utf8');

  // 7. Print the leaderboard.
  printLeaderboard(ranked, criteria);
}
Enter fullscreen mode Exit fullscreen mode

Pair-building is deliberately conservative — a pair is emitted only when
the score gap is meaningful:

export function buildPairs(prompt: string, ranked: ScoredCandidate[], minDelta: number) {
  const top = ranked[0];
  const bottom = ranked[ranked.length - 1];
  const delta = top.score - bottom.score;
  if (delta <= minDelta) return [];   // no meaningful difference → no pair
  return [{ prompt, chosen: top.candidate, rejected: bottom.candidate,
            score_chosen: top.score, score_rejected: bottom.score, score_delta: delta }];
}
Enter fullscreen mode Exit fullscreen mode

DPO dataset format

Each row of data/dataset.jsonl is directly usable by DPO/RLHF training
loops:

{
  "prompt": "Write a hello world",
  "chosen": "Here is a robust hello world: ...",
  "rejected": "I don't know. I cannot answer this question.",
  "score_chosen": 0.842,
  "score_rejected": 0.004,
  "score_delta": 0.838
}
Enter fullscreen mode Exit fullscreen mode

The mock LLM

src/mock.ts is a deterministic stand-in that still exercises the full
logprob path: for every verification call it emits a 20-letter logprob
distribution whose mass concentrates around the "true" quality of the
candidate — so the expectation math runs exactly as it does against a real
endpoint. It also models temperature diversity (low temperature → well-formed
answer, high temperature → evasive), so --mock demos the whole factory
offline.

Unit tests

src/verifier.test.ts pins down the core math with node:test (no
framework). Highlights:

test('expectedScore: degenerate distribution on A → 1.0', () => {
  const row = logprobRow([['A', 0], ['B', -100]]);
  assert.equal(expectedScoreFromLogprobs(row), 1.0);
});

test('expectedScore: uniform over A..T → 0.5', () => {
  const row = logprobRow(LETTERS.map((l) => [l, 0]));
  assert.ok(Math.abs(expectedScoreFromLogprobs(row)! - 0.5) < 1e-12);
});

test('expectedScore: renormalization ignores non-letter tokens', () => {
  // "U" has the highest logprob but is out-of-scale → must be dropped.
  const row = logprobRow([['U', 0], ['A', -1], ['T', -1]]);
  assert.ok(Math.abs(expectedScoreFromLogprobs(row)! - 0.5) < 1e-9);
});
Enter fullscreen mode Exit fullscreen mode

Run with npm test (16 tests, all passing).

Forking & contributing

Get set up 

git clone <your-fork-url>
cd selfplay-factory
npm i
npm test          # 16 tests should pass
npm run typecheck # tsc --noEmit should be clean
Enter fullscreen mode Exit fullscreen mode

If npm i hits a permission error on the global npm cache, use a
project-local cache: npm install --cache "$(pwd)/.npm-cache".

Where to look first

If you want to change… Start here
The scoring math src/verifier.ts + src/verifier.test.ts
The verifier prompt / rubric src/prompts.ts
Candidate generation src/generator.ts
Orchestration / output format src/factory.ts
The HTTP client src/llm.ts
Mock behavior src/mock.ts
CLI flags src/index.ts

Suggested new features (good first contributions)

  1. More preference pairs per prompt. Currently we emit one pair (top vs bottom). Emit pairs from every meaningful gap (top vs each weaker candidate) or a tournament-style selection — more data per prompt.
  2. Semantic dedup. Replace exact-match dedup with embedding-based similarity (or a cheap LLM "is this a duplicate?" check) so near-identical candidates don't inflate the batch.
  3. Self-consistency voting. Ask the verifier for a justification after the letter token and use it to flag low-confidence scores.
  4. Pairwise verifier mode. Instead of absolute letter scores, present two candidates and ask for a preference letter — the classic LLM-as-a-judge setup, but with the same logprob-expectation trick.
  5. Batch / concurrency. scoreCandidates is sequential; parallelize with a bounded-concurrency pool to cut wall-clock time on large batches.
  6. JSONL streaming + resumability. Write rows as they're produced and resume interrupted runs.
  7. Multiple prompt batches. Accept a prompts file and emit one dataset across many prompts (with prompt per row — already in the schema).
  8. Calibration metrics. Add a report of score distribution, inter-criterion agreement, and per-evaluation variance so users can tune --n-evaluations.
  9. More output formats. Support ChatML / ShareGPT / UltraFeedback-style schemas alongside DPO.
  10. Retry + backoff in llm.ts for flaky endpoints, and a --max-retries flag.

Contribution workflow

  1. Fork the repo and create a feature branch.
  2. Add or update unit tests in src/verifier.test.ts (or a new *.test.ts).
  3. Run npm test and npm run typecheck — both must pass.
  4. Open a PR with a clear description and a sample of the output dataset.

Code & more: https://www.dailybuild.xyz/project/233-selfplay-factory

Top comments (0)