A Nightly Model-Regression Harness on Free Models and a Free Server

Last month, a prompt that returned clean JSON for six months started returning markdown. No code changed. No prompt changed. The model changed.

Model drift is the silent production bug. You cannot pin a model version forever, and you cannot trust any provider's "we will let you know." What you can do is run a nightly regression harness that compares today's model output against yesterday's baseline. The catch is cost. A real regression suite burns tokens fast.

Free models change that math. MonkeyCode, an open-source AI coding assistant, offers free model access for experimentation and a free server option for running scheduled jobs. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

This article builds a nightly model-regression harness with three parts: golden prompts, a comparison script, and a scheduler. You will get a decision table for choosing a comparison method, a working Python script, and a list of teams that should not use this approach.

The problem: models change under you

Providers update models without a changelog. They deprecate versions. They tweak system prompts. Your prompt was tuned against a model that no longer exists.

The failure mode is nasty. Your unit tests pass because they mock the model. Your integration tests pass because they use the same model. Production fails because production uses the real model, which changed last Tuesday.

A nightly regression harness catches the drift within 24 hours. It runs a fixed set of prompts, records the outputs, and compares them against a stored baseline. When the diff exceeds a threshold, you get a report before your users get an error.

Part one: golden prompts

Golden prompts are the inputs that matter most. Not your whole prompt library. The twenty prompts that drive revenue-critical paths.

Store them as a JSON file. Each entry has an ID, a prompt, and a baseline output.

[
  {
    "id": "extract-invoice-total",
    "prompt": "Extract the total amount from this invoice. Return JSON with keys: currency, amount.",
    "baseline": "{\"currency\": \"USD\", \"amount\": 1234.56}"
  },
  {
    "id": "classify-ticket",
    "prompt": "Classify this support ticket as billing, technical, or other. Reply with one word.",
    "baseline": "billing"
  }
]

The baseline is captured once, manually, and reviewed. After that, it is frozen. A baseline change is a code change. It needs a PR, not a silent update.

Part two: the comparison script

The script reads the golden prompts, calls the model, and compares each output against the baseline. It writes a report and exits non-zero when the failure rate is too high.

import json
import sys
from difflib import SequenceMatcher

def load_golden(path):
    with open(path) as f:
        return json.load(f)

def call_model(prompt):
    # Replace with your provider's client.
    # MonkeyCode's free model access works here.
    return "{\"currency\": \"USD\", \"amount\": 1234.56}"

def similarity(a, b):
    return SequenceMatcher(None, a, b).ratio()

def main(golden_path, threshold=0.9):
    golden = load_golden(golden_path)
    failures = []

    for case in golden:
        actual = call_model(case["prompt"])
        score = similarity(actual, case["baseline"])
        status = "PASS" if score >= threshold else "FAIL"
        if status == "FAIL":
            failures.append({"id": case["id"], "score": score, "actual": actual})
        print(f"{status} {case['id']}: {score:.2f}")

    if failures:
        with open("regression-report.json", "w") as f:
            json.dump(failures, f, indent=2)
        print(f"{len(failures)} regression(s) detected.")
        sys.exit(1)

if __name__ == "__main__":
    main(sys.argv[1] if len(sys.argv) > 1 else "golden.json")

The script is deliberately simple. No framework, no dependencies beyond the standard library. You can run it anywhere, including the free server option that MonkeyCode provides for scheduled jobs.

Part three: the nightly schedule

A regression harness only works if it runs. Schedule it nightly. A cron job on a free server is enough.

0 2 * * * cd /opt/model-regression && python compare.py golden.json >> nightly.log 2>&1

The free server option matters here. You do not want to pay for a VM just to run a twenty-minute job. A free tier that can run cron jobs is exactly the right size for this workload.

The output lands in a file. The next morning, you check the log. If the script exited non-zero, the report tells you which prompts drifted and how far.

The decision table: how to compare outputs

Similarity is not always the right tool. Choose based on your output type.

Output type	Comparison method	Threshold	Why
Structured JSON	Parse and compare fields	Exact match per field	Order and whitespace do not matter
Single label	Exact string match	100%	One word, no ambiguity
Free-form text	SequenceMatcher or embedding distance	0.85	Semantics matter more than wording
Code	Compile + run tests	Pass/fail	Behavior is the contract
Mixed	Parse first, then similarity on text fields	Per-field	Each field has its own contract

The threshold is a starting point, not a law. Run the harness for a week, look at the scores, and set thresholds that separate real drift from normal variation.

Limitations

This harness does not catch semantic drift. A model can rephrase a response in a way that passes a similarity check but changes the meaning. If your use case is sensitive to meaning, add an LLM-as-judge step. That costs tokens, which is another reason the free model access matters.

The baseline goes stale. Prompts change. Products change. Review the golden set monthly. Remove dead prompts. Add new ones. A golden set that does not evolve becomes a museum.

The harness tests prompts, not pipelines. It will not catch a broken retriever, a changed embedding model, or a bug in your orchestration code. Those need their own tests.

Who should not use this

Teams with no stable prompts. If your prompts change every week, the baseline churns and the report is noise.

Teams that already have a vendor-provided evaluation suite. Use theirs. Do not build a parallel system.

Teams with zero tolerance for false alarms. A nightly regression report that nobody reads is worse than no report. It trains the team to ignore the signal.

Closing

Model drift is real. It is silent. It breaks production without a commit.

A nightly regression harness turns that silence into a report. Golden prompts, a comparison script, and a cron job. That is the whole system. Free models make the token cost manageable. A free server makes the schedule cost zero.

Start with ten prompts. Run it for a week. Read the first report. Then decide if you need more.

The harness is the early warning system. The model is the weather. You cannot control the weather. You can control whether you are surprised by it.