Catch Prompt Drift on a Schedule: Zero-Cost Regression Checks

A support team noticed the chatbot's answers had changed tone overnight. No code changed. No prompt in the repo was touched. Yet the replies sounded stiffer, more defensive.

That's prompt drift: the same inputs producing measurably different outputs, often after a model update or a hidden configuration tweak. When you ship a product on probabilistic systems, you need a way to notice drift before users file complaints about it.

MonkeyCode offers free models and a free server option, which makes this kind of scheduled regression check affordable for a small team. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The idea is simple: store a set of fixed inputs, run them through the model every week, and compare the responses against a recorded baseline. If the behavior shifts beyond a threshold, you get an alert with evidence.

This article walks through building that drift detector with a Python script, a cron job, and MonkeyCode's free infrastructure. No cloud bill, no dedicated ML platform, just a server you already have.

Why a Scheduled Probe Works

Drift detection is not about reading one model output and judging it. It's about collecting a series of measurements over time, then looking for changes in that series.

The cheapest reliable method is a fixed probe suite: a list of representative prompts with expected behavior. Every week, your script sends those prompts to the model, records the responses, and computes a similarity score against the baseline.

A drop in similarity doesn't mean the new behavior is wrong. It means something changed, and a human should investigate. That distinction keeps the alert useful instead of noisy.

What You'll Build

A Python script that does three things:

1. Loads a set of probe prompts from a JSON file.
2. Calls MonkeyCode's model for each probe.
3. Compares each response to a baseline using a hashing method and a small text similarity heuristic.

Then it writes a report to a log file. A cron job runs the script weekly on MonkeyCode's free server.

Step 1: Prepare the Probe Suite

Create a probes.json file. Each probe contains an id, the prompt, and a baseline_hash that you'll generate later. Start with an empty baseline.

{
  "probes": [
    {
      "id": "refund-policy",
      "prompt": "Explain the refund policy in one sentence.",
      "expected_phrase": "within 30 days"
    },
    {
      "id": "greeting-tone",
      "prompt": "Say hello to a new user.",
      "expected_phrase": "help"
    },
    {
      "id": "error-handling",
      "prompt": "What should I do if the upload fails?",
      "expected_phrase": "retry"
    }
  ]
}

You don't need many probes. Ten to twenty prompts covering core user journeys are enough to spot a systemic shift. Too many probes burn tokens without improving detection.

Step 2: Write the Drift Check Script

Create drift_check.py. The script loads the probes, calls the model, and compares the output with the stored baseline.

import os
import json
import hashlib
import requests
from datetime import datetime

API_ENDPOINT = os.environ["MONKEYCODE_ENDPOINT"]
API_KEY = os.environ["MONKEYCODE_API_KEY"]
MODEL = os.environ.get("MONKEYCODE_MODEL", "default")


def call_model(prompt):
    resp = requests.post(
        API_ENDPOINT,
        headers={"Authorization": f"Bearer {API_KEY}"},
        json={
            "model": MODEL,
            "messages": [{"role": "user", "content": prompt}],
            "temperature": 0.0,
        },
        timeout=30,
    )
    resp.raise_for_status()
    data = resp.json()
    return data["choices"][0]["message"]["content"]


def text_hash(text):
    normalized = " ".join(text.lower().split())
    return hashlib.sha256(normalized.encode()).hexdigest()


def similarity(text, expected_phrase):
    return 1.0 if expected_phrase.lower() in text.lower() else 0.0


def load_baseline(path="baseline.json"):
    try:
        with open(path) as f:
            return json.load(f)
    except FileNotFoundError:
        return {}


def save_baseline(data, path="baseline.json"):
    with open(path, "w") as f:
        json.dump(data, f, indent=2)


with open("probes.json") as f:
    probes = json.load(f)["probes"]

baseline = load_baseline()
report = {
    "timestamp": datetime.now().isoformat(),
    "results": []
}

for probe in probes:
    prompt = probe["prompt"]
    output = call_model(prompt)
    current_hash = text_hash(output)

    record = {
        "id": probe["id"],
        "prompt": prompt,
        "output": output,
        "hash": current_hash,
        "expected_phrase_found": similarity(output, probe["expected_phrase"]),
    }

    if probe["id"] in baseline:
        record["drifted"] = current_hash != baseline[probe["id"]]["hash"]
    else:
        record["drifted"] = False
        baseline[probe["id"]] = {"hash": current_hash}

    report["results"].append(record)

# Update baseline only if this run succeeds
save_baseline(baseline)

with open(f"drift_report_{report['timestamp'][:10]}.json", "w") as f:
    json.dump(report, f, indent=2)

print(f"Checked {len(probes)} probes. Drifted: {sum(1 for r in report['results'] if r['drifted'])}")

This version stores the exact hash of the output. If the model's response changes even slightly, the hash changes and marks the probe as drifted. That's strict, but it catches small changes that a similarity metric might miss.

For a looser comparison, replace the hash check with a semantic similarity call, but that costs extra tokens. Start strict and relax only when false alarms become a problem.

Step 3: Generate Your First Baseline

Run the script locally or directly on the server.

pip install requests
python drift_check.py

On the first run, there's no baseline yet, so every probe is marked drifted: false. The script creates baseline.json with the current hashes.

Check one report file:

cat drift_report_*.json | python -m json.tool

Every result should include the output text, the hash, and whether the expected phrase appeared. If any probe is missing its expected phrase, fix the prompt or the expected phrase before trusting the baseline.

Step 4: Deploy to MonkeyCode's Free Server

MonkeyCode's free server is a small Linux box. Copy your files there and set up the environment.

scp probes.json drift_check.py user@your-free-server:~/
ssh user@your-free-server

Add your API credentials to ~/.bashrc so they persist across sessions.

echo 'export MONKEYCODE_API_KEY="your-key-here"' >> ~/.bashrc
echo 'export MONKEYCODE_ENDPOINT="<endpoint>"' >> ~/.bashrc
echo 'export MONKEYCODE_MODEL="default"' >> ~/.bashrc
source ~/.bashrc

Run the script once to confirm it works on the server:

python drift_check.py

Step 5: Schedule Weekly Checks

Open the crontab:

crontab -e

Add a line to run every Monday at 6 AM:

0 6 * * 1 cd ~ && python drift_check.py >> drift_cron.log 2>&1

Verify after the first scheduled run:

cat drift_cron.log

You should see the summary line with the number of drifted probes. If you set up email or a webhook, you can route that output to your team's channel. But even without notifications, the JSON report is your audit trail.

What to Do When Drift Fires

When a probe drifts, don't panic. First compare the current output with the baseline output. Read the two side by side:

python -c "import json;r=json.load(open('drift_report_YYYY-MM-DD.json'));[print(x['id'], x['drifted'], x['output'], '\n') for x in r['results']]"

Ask three questions:

1. Is the new output factually wrong, or just differently worded?
2. Does the expected phrase still appear? If yes, the drift is likely stylistic.
3. Did any dependency or model version change during that week?

If the drift is material, update the baseline and tell the team. If it's noise, the hash threshold is too strict. Consider using a fuzzy match on the expected phrase rather than the whole output hash.

Limitations

This is a lightweight signal, not a full evaluation suite. It can't detect importance-weighted drift or nuanced semantic changes. It also assumes you have a stable set of probes; if your product changes weekly, the probes become stale.

The free model and free server are best for low-volume checks. Running hundreds of probes every day will exhaust quotas. Keep the suite small and the cadence weekly.

Also, the default model name is a placeholder. Check your MonkeyCode dashboard for the exact model identifier and replace it in the environment variable.

Who Should Not Use This

Skip this approach if you need real-time drift detection or if your team already runs a production monitoring stack with vectors and alerting. This is for teams that have no drift detection today and want a cheap, reproducible starting point.

If you're running legally sensitive responses, don't rely on a weekly hash drift check. Use a formal evaluation harness with human review.

The Next Step

Start with five probes covering your most important user flows. Run the script for three weeks. Then read the accumulated reports and ask whether the model's behavior stayed stable.

That small habit turns drift from a surprise into a measured event. You'll know when something changed, and you'll have the receipts to show why it matters.