The Same 11 Prompts, Three Copies, 48 Hours: A Free Model Regression Battery

The same model read the same prompt three times in one hour and proposed three different action plans, each one delivered with the same tone of confidence. That was the moment I stopped treating the free model like a pure function and started reading it like a sensor: the signal exists, but only the statistics are worth trusting.

I wanted to know whether my extraction code could survive a 48-hour regression run when the upstream model could not hold a single version of the truth. The answer was not more assertions; it was freezing an 11-prompt battery, recording every raw output, and refusing to tune anything mid-flight.

What I set up

I froze a battery of 11 prompts, ranging from single-key JSON requests to multi-paragraph summaries. The code below shows five of them because the other six are embarrassingly specific to my own failure log, but the shape of the loop is what matters.

import hashlib
import json
import time
from collections import defaultdict
from pathlib import Path

PROMPTS = [
    "Return JSON with keys: status, reason.",
    "Summarize this function in one line.",
    "Pick the safest option and explain why.",
    "Rewrite this error message for a new developer.",
    "Write a commit message for this diff.",
    # ...plus six more from my own failure log
]

def run_battery(call_model, out_path: Path, copies: int = 3):
    rows = []
    for prompt in PROMPTS:
        for copy in range(copies):
            text = call_model(prompt)
            rows.append({
                "prompt_sha": hashlib.sha256(prompt.encode()).hexdigest()[:8],
                "copy": copy,
                "length": len(text),
                "text_sha": hashlib.sha256(text.encode()).hexdigest()[:16],
                "head": text[:120].replace("\n", " "),
                "ts": time.time(),
            })
            time.sleep(1)
    out_path.write_text(json.dumps(rows, indent=2))

def diversity_ratio(rows):
    by_prompt = defaultdict(list)
    for row in rows:
        by_prompt[row["prompt_sha"]].append(row)
    return {
        sha: round(len({r["text_sha"] for r in group}) / len(group), 2)
        for sha, group in by_prompt.items()
    }

With three copies per prompt, a perfect repeat scores 0.33 and a fully unstable output scores 1.0. A score between those two values usually meant one copy wandered off while the others agreed, which was already enough to break my parser.

Every six hours I ran the battery against the same model route and stored the JSON output as a separate file. I used MonkeyCode's free model access for the calls and hosted the loop on its free server option, mostly because I did not want to keep my laptop awake for two nights. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The same experiment would work with any similar API, and the lesson is not tied to that specific provider.

What broke

My first assumption was that short prompts would behave and long prompts would wander. That turned out to be backwards: short prompts were stable for a while, then flipped an entire key name with no warning, while long prompts showed high diversity from the first run and never improved.

• The flakiest part was not the model; it was my assertion layer. One run returned reason, the next returned explanation, and my code had no idea they were the same thing. The text_sha hash changed, which was honest, but the real failure was an output contract that I had defined too narrowly.
• JSON requests came back in three shapes: raw JSON, fenced code blocks, and pretty-printed text with a preamble. Same prompt, same route, same hour, three parse paths. I stopped calling that model drift; it is serialization drift, and it eats more debugging time than any semantic difference.
• The diversity ratio was only useful at the extremes. Values near 0.33 meant agreement, values near 1.0 meant noise, and everything in between was too small a sample to interpret. Three copies gave variance, not confidence; by hour 30 I decided to trust only the two ends of the scale.
• I caught myself helping the experiment. At hour 14 I wanted to rephrase one prompt because the output looked confusing, and that would have silently created a completely different experiment. I kept a change log anyway, because at hour 40 the memory of what I changed is gone.

What I would repeat

1. Freeze the prompt list fiercely. One edit means an entirely new battery; the old runs become useless for comparison.
2. Keep three cheap assertions everywhere: substring present, key present, length inside a window. Anything smarter becomes a second model that also needs testing.
3. Store raw text and hash it to JSONL, and do not print pretty output during the run. Human reading is how false patterns sneak into the notes.
4. Drop any prompt that shows 1.0 twice in a row. It is not giving a regression signal; it is giving ambient noise.
5. Write down exactly when and why you changed something, even if the change feels trivial. Two days later, every detail looks foreign.

Who should not use this

If you need correctness rather than repeatability, this battery will disappoint you. It measures how stable an output is, not which version of the output is right, so it suits extraction pipelines and alerting experiments much better than production decisions.

If every response must fit a strict schema, budget for a real validation layer: schema checks, retries, and human review. A free model over 48 hours will reliably produce the one shape you forgot to handle.

The part I keep

After two days I did not get a model with better memory. I got a table of numbers that told me which prompts deserved automation and which ones needed a human in the loop, and that is worth more than a test suite pretending outputs are stable.

My next 48-hour experiment will start the same way: script on MonkeyCode's free server option, calls routed through its free model access, JSON pulled out, machine deleted. What remains should be logs, not chat history, because a cheap repeatable experiment is only useful when it is also easy to throw away.