48-Hour Field Notes: The Prompt That Broke Without a Changelog

A prompt that returned clean JSON on Friday came back on Monday as a friendly sentence wrapped around the JSON. No changelog. No version bump in my config. The endpoint string was byte-for-byte identical, and that is exactly the problem: an alias is not a version.

So I spent 48 hours on one narrow question — can a solo developer notice a model swap before a user does, using nothing but a cron job and a git repo? Short answer: yes, if you stop diffing text and start tracking a small, stable set of prompt digests.

Hour 0-4: why diffing the raw output fails

My first instinct was the obvious one. Store yesterday's answers, compare today's, print the difference. That lasted about ninety minutes. An answer that says "30 seconds" one day and "30.0 seconds" the next is not a regression, and a diff tool will scream at both that and a genuine format break with identical volume.

The fix is normalization before hashing: collapse digits to a placeholder, squeeze horizontal whitespace, collapse blank-line runs, lowercase. You lose the ability to see what changed in the digest, but you gain a signal that only fires when the shape of the answer moves. Keep the raw outputs too, on disk, for the day the digest actually flips.

#!/usr/bin/env python3
"""canary.py - detect silent model drift on an OpenAI-compatible endpoint."""
import hashlib, json, os, re, urllib.request

BASE = os.environ["LLM_BASE_URL"].rstrip("/")
KEY = os.environ["LLM_API_KEY"]
MODEL = os.environ["LLM_MODEL"]
BASELINE = "baseline.json"
RUNS = 2  # two reads per prompt: separates flake from a real swap

NUM = re.compile(r"\d+(?:\.\d+)?")
WS = re.compile(r"[ \t]+")

def call(prompt: str) -> str:
    body = json.dumps({
        "model": MODEL,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0,
    }).encode()
    req = urllib.request.Request(
        f"{BASE}/chat/completions",
        data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {KEY}"},
    )
    with urllib.request.urlopen(req, timeout=90) as resp:
        return json.loads(resp.read())["choices"][0]["message"]["content"]

def norm(text: str) -> str:
    text = NUM.sub("<n>", text)
    text = WS.sub(" ", text)
    text = re.sub(r"\n{2,}", "\n", text)
    return text.strip().lower()

def digest(text: str) -> str:
    return hashlib.sha256(norm(text).encode()).hexdigest()[:16]

def main() -> int:
    prompts = json.load(open("prompts.json"))
    baseline = json.load(open(BASELINE)) if os.path.exists(BASELINE) else {}
    report, drift = {}, []

    for pid, prompt in prompts.items():
        seen = [digest(call(prompt)) for _ in range(RUNS)]
        stable = seen[0] == seen[1]
        known = baseline.get(pid)
        if known is None:
            verdict = "new"
        elif not stable:
            verdict = "flaky"
        elif seen[0] == known:
            verdict = "stable"
        else:
            verdict = "drift"
        report[pid] = {"digests": seen, "baseline": known, "verdict": verdict}
        if verdict == "drift":
            drift.append(pid)

    os.makedirs("runs", exist_ok=True)
    run_id = os.environ.get("RUN_ID", "local")
    json.dump(report, open(f"runs/{run_id}.json", "w"), indent=2)
    print(json.dumps({k: v["verdict"] for k, v in report.items()}, indent=2))
    return 1 if drift else 0

if __name__ == "__main__":
    raise SystemExit(main())

Two details earn their keep. temperature: 0 reduces variance but does not remove it, which is why every prompt is read twice. And the exit code is the whole monitoring story: 1 means something moved, 0 means nothing did, and cron does not care about your feelings.

Hour 4-12: choosing prompts that fail loudly

A canary suite is only as good as its worst prompt. I picked three classes, and three was enough. One prompt demands a strict JSON object with fixed keys, because schema drift is the failure that actually breaks a parser in production. One asks for a one-sentence summary of a changelog entry, because refusal wording and verbosity drift show up there first. One asks for arithmetic with the steps shown, because a swap toward a smaller or differently tuned model usually changes the shape of the reasoning before it changes the answer.

{
  "format_json": "Return only a JSON object with keys id, severity, owner for this log line: ERROR db timeout after 30s",
  "summary_floor": "Summarize in exactly one sentence: deprecate /v1/charge, add /v2/charge with same fields.",
  "arith_shape": "A queue is 12000 deep and draining at 40/s. Time to empty? Show the arithmetic."
}

Run it once to seed the baseline, commit baseline.json next to the script, and never edit it by hand without a reason in the commit message. That file is the contract.

Hour 12-36: the run that has to survive a recycled box

Here is where an ephemeral free server changes the design. You cannot trust local disk to hold your history, so the repo has to be the memory. The cron line writes the run file, commits it, and pushes before the box can be reclaimed.

# every six hours, from the free server
17 */6 * * * cd ~/canary && \
  RUN_ID=$(date -u +%Y-%m-%dT%H) python3 canary.py >> runs.log 2>&1; \
  git add runs baseline.json && \
  git commit -qm "canary $RUN_ID" && git push -q origin main

What you get back is a shape, not a screenshot. The output below is an illustrative template run — the verdict strings and exit code are real behavior of the script, but these particular digests are placeholders, so do not read them as measurements of any provider.

{
  "format_json": "drift",
  "summary_floor": "stable",
  "arith_shape": "flaky"
}

That combination is the useful part. One prompt moved and held its new shape across both reads, which means the endpoint's behavior changed. A second flipped between reads, which means sampling noise and nothing else. If every prompt drifts at once, you are not chasing a model change, you are chasing a re-pointed alias, and the right move is to re-baseline and eyeball three raw samples instead of filing a bug.

Symptom	Likely cause	First move
format_json drifts, prose stable	output template or schema change	pin the contract, add one parse-and-retry
every prompt drifts together	alias re-pointed to a new model	re-baseline, diff three raw samples by hand
one prompt flips between reads	sampling, not a swap	raise RUNS, do not page anyone
everything reports new	baseline never seeded or lost	commit baseline.json, re-run

Hour 36-48: where the free tier actually fits

I ran the suite against free model access and let cron live on a free server option, which is the only reason a 48-hour experiment like this is affordable at all. The economics are simple: a canary suite is small, frequent, and boring, and boring traffic is exactly what you should not be paying metered rates for. I used MonkeyCode for both halves — the free model access behind the harness, and the free server holding the cron entry. I did not pin any specific model name in this article on purpose, because the harness only needs a base URL, a key, and a model string from the environment. The operator advertises a free allowance on the order of 10 million tokens plus the free server option; treat that as a claim to verify on the project page at the moment you sign up, since free-tier terms move.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The honest limitation: this workflow measures change, not correctness. A model can drift into a better answer and trip your alarm, and a model can degrade in a way your three prompts never touch. It is a smoke detector, not a unit test suite. Keep the prompt count small enough that a human reads every verdict.

Who should skip it: anyone whose output is contractually bound to a named model version with a vendor SLA, where the right instrument is the vendor's changelog and not a cron job. And anyone who has not yet written down what the output is supposed to look like — a canary with no expectation just records weather.

What I would repeat in the next 48 hours is the boring part. Normalize before hashing. Read twice before alarming. Commit the results off the box. If you run this against a free endpoint, the interesting question is not whether it drifts, but which of the three prompt classes flips first for you.