Free Models, Fixed Probes: Catch Prompt Drift Before It Reaches Users

A product manager changed one word in the system prompt. "Helpful" became "concise." The agent stopped answering follow-up questions. It repeated its first answer and ignored everything after.

Nobody noticed until a customer did.

The customer asked a clarifying question. The agent gave the same answer twice. The customer left. The team spent a week debugging a bug that was never a bug. It was a prompt change.

This is prompt drift. A small edit shifts behavior in ways nobody intends. The shift is invisible in a single test. It only appears when you compare behavior before and after.

Most teams compare nothing. They update the prompt, run one happy-path test, and ship. The one happy path passes. The other twenty behaviors changed silently.

The fix is a probe set: a fixed list of inputs with fixed expectations about behavior. You run it before a prompt change, run it after, and diff the results. The diff is your drift report.

Probe sets only work if you actually run them on every change. That is a cost problem. Every run burns tokens. Teams skip runs to save money, and drift slips through.

MonkeyCode is an open-source agent platform that offers free models and a free server option, which removes that excuse. Free models make the per-run cost zero. The free server hosts the agent so the probe run is a single API call. This article shows how to build a probe set and use that free tier to catch drift before users do.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What a probe set checks

A probe set does not check whether the answer is correct. It checks whether the agent still behaves the same way.

That distinction matters. Correctness is hard to assert automatically. Behavior is easy.

A probe has three parts:

1. A prompt, fixed forever.
2. Words that must appear in the reply.
3. Words that must not appear.

For a support agent, a probe might assert that the reply contains the word "policy" and does not contain the phrase "I can only." For a coding agent, a probe might assert that the reply contains a code block and does not contain the word "sorry."

The assertions are crude. That is intentional. Crude assertions catch large behavioral shifts, and large shifts are what cause customer-facing incidents.

Step 1: Define the probe file

Start with ten probes. Ten is enough to catch the common drift patterns: refusal creep, verbosity collapse, and context loss.

Here is a probe file for a fictional support agent:

[
  {
    "id": "greeting",
    "prompt": "Hi, I need help.",
    "expect": {
      "must_include": ["hi", "help"],
      "must_not_include": ["cannot"]
    }
  },
  {
    "id": "followup",
    "prompt": "Thanks. Can you also check my second order?",
    "expect": {
      "must_include": ["second"],
      "must_not_include": ["first order only"]
    }
  },
  {
    "id": "refund",
    "prompt": "What is your refund policy?",
    "expect": {
      "must_include": ["policy"],
      "must_not_include": ["contact support"]
    }
  }
]

The third probe catches the classic drift pattern: the agent stops answering and starts deflecting. The second probe catches context loss. The first catches refusal creep.

Save it as probes.json. Add seven more probes that match your own product's critical flows.

Step 2: Write the probe runner

The runner sends each probe to the agent and checks the reply against the assertions.

import json
import os
import sys
from openai import OpenAI

client = OpenAI(
    base_url=os.environ["MONKEY_BASE_URL"],
    api_key=os.environ["MONKEY_API_KEY"],
)

SYSTEM_PROMPT = os.environ.get("SYSTEM_PROMPT", "You are a support agent.")

def run_probe(probe):
    response = client.chat.completions.create(
        model=os.environ["MONKEY_MODEL"],
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": probe["prompt"]},
        ],
    )
    return response.choices[0].message.content

def check(probe, reply):
    text = reply.lower()
    missing = [m for m in probe["expect"].get("must_include", []) if m not in text]
    forbidden = [m for m in probe["expect"].get("must_not_include", []) if m in text]
    return missing, forbidden

def main(probe_file):
    with open(probe_file) as f:
        probes = json.load(f)
    results = []
    for probe in probes:
        reply = run_probe(probe)
        missing, forbidden = check(probe, reply)
        passed = not missing and not forbidden
        results.append({"id": probe["id"], "passed": passed})
        print(f"{'PASS' if passed else 'FAIL'} {probe['id']} missing={missing} forbidden={forbidden}")
    return all(r["passed"] for r in results)

if __name__ == "__main__":
    sys.exit(0 if main(sys.argv[1]) else 1)

Save it as probe.py. Install the client:

pip install openai

Set the environment variables. The free server provides the base URL and key. The dashboard lists the model id.

export MONKEY_BASE_URL="https://your-free-server.example"
export MONKEY_API_KEY="sk-..."
export MONKEY_MODEL="your-model-id"

Run the baseline:

python probe.py probes.json

Every probe should pass. If one fails, fix the probe before continuing. A failing baseline makes the drift report meaningless.

Step 3: Capture the baseline

The baseline is your reference point. Store the output:

python probe.py probes.json > baseline.txt
cat baseline.txt

You expect ten PASS lines. This file is the behavior contract. It says: on this date, with this system prompt, the agent behaved this way.

Keep it in version control next to the prompt file. When the prompt changes, the diff tells the story.

Step 4: Run after a prompt change

Now edit the system prompt. Change one word, add a sentence, reorder instructions. Then run again:

python probe.py probes.json > after.txt
diff baseline.txt after.txt

The diff is the drift report. Every line that changed is a behavior shift. Some shifts are improvements. Some are regressions. The diff does not judge. It only reports.

Step 5: Classify with a decision table

A probe result can change in four ways. Each way has a different action.

Baseline	After change	Classification	Action
PASS	PASS	No drift	Ship it
PASS	FAIL	Regression	Roll back or fix
FAIL	PASS	Improvement	Update baseline
FAIL	FAIL	Unrelated	Check server and prompt

The middle two rows are where teams make mistakes. An improvement looks like a failure if you only look at the after state. A regression looks harmless if the probe was already failing.

The table forces a decision. No row says "maybe."

Who should not use this

Probe sets assume stable, repeatable flows. If your agent handles open-ended creative work, a fixed probe set will mislead you. A poem generator has no "must_include" words that survive a style change.

Probe sets also miss what they do not cover. Ten probes will not catch a drift in a flow you never wrote down. The probe set is a smoke alarm, not a security camera.

Free models have limits too. They are not the strongest models available, and a weak model may fail probes for the wrong reason. A FAIL might mean the prompt drifted, or it might mean the free model could not follow the new instruction. Check the reply text before blaming the prompt.

The cost math

A probe run is ten small requests. On a paid API, that cost adds up if you run it fifty times a day. On the free tier, it is zero. The only cost is the five minutes it takes to read the diff.

That is the point. Drift detection only works when it happens every time. Free models make every-time affordable. The free server makes it a single curl away.

Start with ten probes. Store the baseline. Run the diff on the next prompt change. The one-word edit that breaks behavior will show up as a red line instead of a lost customer.