DEV Community

Kiran Kumar S D
Kiran Kumar S D

Posted on AI-assisted

Pin your LLM eval dataset before comparing two prompts

Picture a familiar scene. You tweak a prompt, run it against your test questions, and the pass rate goes up. You post it in the team channel. Then someone asks, "Did you run both prompts on the same examples?"

Maybe you did. Maybe the dataset changed between runs because someone added cases, a script resampled the data, or a retrieval index updated. If you can't say for sure, your comparison doesn't mean much.

This article is about one habit that prevents that: pin your eval dataset before you compare two prompts.

The scenarios below are illustrative, not from a real project.

Quick definitions

An eval dataset is a set of inputs, and often expected outputs or grading criteria, that you use to measure how well your LLM feature behaves.

Pinning means freezing a specific, versioned snapshot of that dataset so it can't change underneath you. Same cases, same order, same content, every run.

A prompt comparison is running prompt A and prompt B against the same dataset and judging which performs better.

The problem: you're comparing two things at once

A prompt comparison is only a clean experiment if exactly one thing changes: the prompt. If the dataset also moves, you can't tell which change caused the difference.

Here are common ways datasets drift without anyone noticing:

  • Someone appends new test cases to a shared file.
  • A script samples 50 random cases on each run, with no fixed seed.
  • Expected outputs get corrected for a few rows mid-week.
  • Test inputs are pulled from a live source, like a database or a docs site.
  • Cases are filtered differently because a config flag changed.

Any of these can shift your score. LLM outputs already vary from run to run, so you have enough noise without adding more.

A hypothetical example

Say you're building a support-ticket classifier. Prompt A is your current one. Prompt B adds a short list of category definitions.

On Monday you run prompt A on tickets.jsonl and get 82% correct. On Wednesday a teammate adds 30 hard, ambiguous tickets to that file. You run prompt B and get 78%.

Is B worse? Possibly. But B was graded on a harder test. You can't conclude anything.

If you had pinned the dataset, both runs would use the same Monday snapshot, and the 30 new tickets would go into a new version for the next round.

How to pin a dataset in practice

You don't need special tooling. Start simple.

1. Version the file

Keep the dataset in git and reference a specific commit or tag. Or name files by version: tickets_v1.jsonl, tickets_v2.jsonl. Never edit a version after you've used it for a comparison. Create a new one.

2. Record a hash

A hash is a short fingerprint of a file's contents. If the file changes by even one character, the hash changes. Here's a small Python example:

import hashlib

def dataset_hash(path: str) -> str:
    h = hashlib.sha256()
    with open(path, "rb") as f:
        h.update(f.read())
    return h.hexdigest()[:12]
Enter fullscreen mode Exit fullscreen mode

Log this hash with every eval run. When two results show different hashes, you know not to compare them directly.

3. Fix any sampling

If you must use a subset, sample once and save the result. If you sample at run time, use a fixed seed:

import random

random.seed(42)
subset = random.sample(cases, 50)
Enter fullscreen mode Exit fullscreen mode

Saving the subset to a file is safer than relying on a seed, because library behavior can change.

4. Store run metadata

Each eval run should write a small record:

{
  "prompt_id": "classifier_prompt_b",
  "dataset_file": "tickets_v1.jsonl",
  "dataset_hash": "a3f9c21d07be",
  "model": "your-model-name",
  "temperature": 0,
  "run_at": "2025-01-15T10:30:00Z"
}
Enter fullscreen mode Exit fullscreen mode

Now anyone can check whether two results are comparable in seconds.

5. Fail loudly on a mismatch

Add a guard to your comparison script:

if run_a["dataset_hash"] != run_b["dataset_hash"]:
    raise ValueError("Dataset mismatch: results are not comparable.")
Enter fullscreen mode Exit fullscreen mode

A guard like this turns a silent mistake into an obvious one.

Pinning other inputs that act like data

Prompts often depend on more than the test file. If your feature uses retrieval, the document index is part of the dataset in practice. If your prompt includes dynamic context, like the current date or user profile, freeze those values for the eval.

Also pin the model version and settings. If the provider updates a model alias between runs, you've changed another variable. Use a dated model identifier where one is offered, and set temperature low for comparisons.

Pinned doesn't mean permanent

A frozen dataset can go stale. Real usage changes, and your test set should follow it.

The fix is to treat the dataset like a release. When you add cases, publish v2. Then re-run your current best prompt on v2 to get a new baseline. Compare new candidates against that baseline, not against old scores from v1.

A QA lens

Quality engineers already know this rule from regression testing: you don't change the test and the code under test at the same time and then trust the result. Prompts are the code under test here. The dataset is your test suite. Keep one stable while you change the other.

For engineering leaders, the takeaway is simple. When someone reports that a prompt improved, ask for the dataset version and hash alongside the score. It's a small question that filters out a lot of false confidence.

A short checklist

  • Dataset stored with a version, never edited in place
  • Hash logged on every run
  • Sampling fixed or saved
  • Model version, temperature, and context pinned
  • Comparison script refuses mismatched datasets
  • New cases go into a new version, with a fresh baseline

Wrapping up

Better prompts matter, but trustworthy comparisons matter first. Pinning the dataset costs an hour to set up and saves you from arguing about numbers that were never comparable.

What does your team do today to make sure two eval runs used the same data? I'd like to hear what's working, or where it quietly breaks.

Top comments (0)