DEV Community

Avery Lin
Avery Lin

Posted on

Pin Your AI Model Like a Dependency: A Model Lockfile for Reproducible Code Generation

#testing #devops #ai #programming

Your CI pipeline can fail overnight without a single source change. The code is identical, the test suite is identical, and yet the AI-generated patch that passed on Friday no longer passes on Monday. The usual suspect is a floating model: the provider updated the weights or changed the inference behavior behind the same API name, and your prompt now produces subtly different output.

Most teams would never deploy with an unpinned dependency, but they still call model='gpt-4' or a generic free-model endpoint and expect reproducible results. A model is a dependency. Treat it like one.

The artifact: a model lockfile

Instead of recording only which model name you used, record the inputs that produced the output. Here is a minimal model-lock.json:

{
  "schema": "model-lock/v1",
  "model": {
    "provider": "your-provider",
    "name": "free-model-7b",
    "revision": "2026-08-01",
    "temperature": 0.2,
    "max_tokens": 1024
  },
  "inputs": {
    "prompt_template_hash": "sha256:9f2c4d8e...",
    "test_suite_hash": "sha256:c71a9b02..."
  },
  "evidence": {
    "generated_at": "2026-08-14T08:00:00Z",
    "output_hash": "sha256:beef1234...",
    "tests_passed": 17,
    "tests_total": 17
  }
}
Enter fullscreen mode Exit fullscreen mode

The two input hashes matter more than the model name. If either changes, the previous evidence no longer applies to the current generation path.

A small drift-check script

The script below is deliberately small so you can read it in one pass; treat it as a starting check, not a security boundary.

import hashlib
import json
import sys
from pathlib import Path


def sha256_file(path: Path) -> str:
    return "sha256:" + hashlib.sha256(path.read_bytes()).hexdigest()[:12]


def sha256_text(text: str) -> str:
    return "sha256:" + hashlib.sha256(text.encode()).hexdigest()[:12]


def check_drift(prompt_file: Path, tests_dir: Path, lock_file: Path) -> None:
    lock = json.loads(lock_file.read_text())
    prompt_hash = sha256_file(prompt_file)
    test_hash = sha256_text(
        "".join(p.read_text() for p in sorted(tests_dir.rglob("*.py")))
    )

    failures = []
    if prompt_hash != lock["inputs"]["prompt_template_hash"]:
        failures.append(
            f"prompt template drift: {lock['inputs']['prompt_template_hash']} -> {prompt_hash}"
        )
    if test_hash != lock["inputs"]["test_suite_hash"]:
        failures.append(
            f"test suite drift: {lock['inputs']['test_suite_hash']} -> {test_hash}"
        )

    if failures:
        print("Drift detected. Re-evaluate the model before trusting generated code.")
        for failure in failures:
            print(" -", failure)
        sys.exit(1)

    print(f"No input drift. Lockfile {lock_file} is consistent.")


if __name__ == "__main__":
    check_drift(Path("prompts/gen.prompt"), Path("tests"), Path("model-lock.json"))
Enter fullscreen mode Exit fullscreen mode

Run it in CI before every AI-assisted step. If the prompt or test suite moved but the lockfile wasn't regenerated, the step fails.

Where free model access and a free server fit

The main reason people skip this is infrastructure. Running multiple model evaluations usually requires a GPU box or paid API credits. To keep the evaluation cheap, you can run it against a free server. MonkeyCode's operator says the platform provides free model access and a free-server option; that removes the "I need a GPU before I can test model drift" excuse. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The lockfile itself is vendor-neutral, so the same check works against any OpenAI-compatible endpoint.

The transport doesn't matter for the lockfile. What matters is that you can afford to re-run the prompt and tests when you change the model, the prompt, or the test suite.

Use the lockfile to compare models without guessing

A free model is not a single fixed artifact either. If you try two different free models, record both outputs and test pass counts in an alternatives array. Then you can promote a candidate based on evidence instead of memory.

A practical comparison pass looks like this:

  1. Freeze the prompt template and test suite.
  2. Compute their hashes once.
  3. Generate a patch with candidate A, run tests, store output hash and pass count.
  4. Generate a patch with candidate B, run tests, store the same.
  5. Regenerate the lockfile before merge and commit it next to the prompt file.

This turns a vague "the free model feels better this week" into a reproducible record.

Limitations

  • Provider revision strings are not always exposed or stable. A name like free-model-7b may point to different weights over time even if you didn't change anything.
  • A hash only proves the input bytes didn't change. It doesn't prove the output is correct, safe, or production-ready.
  • Free tiers can change quotas, availability, and retention. Don't design a permanent production gate around a free endpoint without a fallback.
  • Hashing prompt templates and test suites won't catch drift in the model's internal behavior if the provider doesn't report it; you still need re-runs on a schedule.

Who should not use this

  • If you're sending proprietary code or secrets to a third-party free endpoint, resolve the compliance boundary first. The lockfile doesn't fix data exposure.
  • If your generation task is open-ended creative writing or exploration, hashing prompts and tests may add noise without much benefit.
  • If you already use a model endpoint with immutable, documented versions and you re-run evals on every release, a full drift-check script may be overkill.

The smallest useful habit

Keep the model-lock.json next to your prompt file. The next time CI fails mysteriously, you'll know whether the model moved under you before you blame the code.

Top comments (0)