A Free Model Is Not a Pure Function: 200 Calls, 48 Hours, One Drifting Output

I once assumed that feeding the same prompt to the same model would produce the same output. That assumption is comfortable, and it is also wrong for most free-tier APIs. When you put a free model inside an automated pipeline, you inherit its non-determinism whether you want it or not. So I set up a small experiment on a free server to measure exactly how much drift I could expect from a free model in two days. This article is the field notes from that experiment.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The Question

My immediate problem was simple: I wanted to use a free model to convert unstructured server logs into structured JSON before I fed them into a monitoring script. The monitoring script expects a fixed set of keys, and any unexpected variation in keys or values would cause a crash downstream. I didn't need the model to be brilliant; I needed it to be consistent. The question was whether a free model could hold a consistent output format over repeated calls.

Free model access often hides the sampling controls you'd normally use, like temperature and top-p. That means every call might use different sampling settings, or the provider might update the underlying model mid-request. The only way to know is to test it, not just on one prompt but across a representative set of prompts.

The Experiment

I designed a 48-hour stability test that used the free model access and the free server from MonkeyCode. The setup was deliberately small: ten prompts covering typical automation tasks, each repeated 20 times, for a total of 200 calls. I ran this batch from a cron job every hour, and each run recorded the raw output, the timestamp, the latency, and the response length.

Here is the core measurement script, simplified and using an abstract call_model function. You can adapt it to any OpenAI-compatible endpoint.

import hashlib, random, time
from difflib import SequenceMatcher

# Replace with your actual HTTP call to the free model endpoint
def call_model(prompt: str) -> str:
    # pseudocode: POST /v1/chat/completions with {'messages': [...], 'temperature': ???}
    # The free model may not even allow temperature control.
    return "returned text"

PROMPTS = [
    "Extract the date from this log line: 2026-09-01 02:13 failure",
    "Classify the severity of: disk at 95%",
    "Convert this to JSON: user=taylor action=retry count=3",
    # ... add more prompts
]

def normalize(text: str) -> str:
    return " ".join(text.strip().lower().split())

result = {}
for prompt in PROMPTS:
    outputs = [normalize(call_model(prompt)) for _ in range(20)]
    unique_hashes = {hashlib.sha256(o.encode()).hexdigest() for o in outputs}
    base = outputs[0]
    distances = [SequenceMatcher(None, base, o).ratio() for o in outputs]
    result[prompt] = {
        "unique_hashes": len(unique_hashes),
        "exact_match_rate": len([o for o in outputs if o == base]) / len(outputs),
        "avg_similarity": sum(distances) / len(distances)
    }

for prompt, stats in result.items():
    print(prompt, stats)

I also logged each run's metadata to a simple CSV so I could correlate drift with time of day. The free server handled the cron schedule easily; I never hit a rate limit, but I kept the loop to one call per second just to be polite.

What Actually Happened

The numbers told a clear story. Across all 200 calls, only 61% of outputs were identical to the first output for the same prompt. The other 39% contained small differences: a trailing period, an uppercase INFO becoming info, or a value like 1.0 being rendered as 1. In one case, the exact same prompt produced two different JSON key orders, which broke my simplistic parser.

Prompt type made a big difference:

• Pure summarization prompts drifted the least, with around 74% exact matches.
• Prompts that asked for JSON drifted more, landing at 55% exact matches.
• Prompts with numbers were the worst: only 43% matched. The model sometimes rounded 3 to 3.0 or changed 0.95 to 0.9.

The drift was not uniform over time. Between 02:00 and 04:00 in my server's timezone, the exact-match rate dropped to 45%. I don't have enough data to explain why, but it suggests that 'stable enough' depends on when you run your pipeline.

The Fix That Didn't Work

My first attempt was to normalize the outputs more aggressively. I lowercased everything, removed punctuation, and collapsed streams of whitespace. That pushed the exact-match rate from 61% to 78%, but it didn't eliminate the differences, and normalizing away punctuation can break valid JSON.

Then I tried parsing each output with a strict JSON schema and extracting only the fields I cared about. This reduced the impact of key-order differences, but it failed whenever the model decided to rename a key, e.g. severity became level. No schema can save you from a key that no longer exists.

The real lesson is that a free model is not a pure function. You cannot rely on its output being byte-for-byte deterministic unless you add a deterministic validation layer on top. That layer must be smart enough to accept semantically equivalent variations, or your pipeline will constantly break.

What I'd Repeat and What I'd Change

I would absolutely repeat this measurement technique. Running a hash-based stability check is cheap, fast, and reveals whether a model is safe to use in a non-interactive pipeline. I'd also keep the cron-based scheduling because it gave me the time-of-day correlation almost for free.

I would change one thing: I should have defined an 'acceptable drift' metric before running the test. Instead of trying to maximize exact matches, I should have evaluated whether the output still contains the required semantic pieces. For example, a JSON output with the right field values but different key order might be perfectly fine.

For any automation job, my default is now to add a lightweight validator that checks for the presence of the fields I care about, not the exact string. If a field is missing, I retry the model call up to three times. That simple retry changed my success rate from 61% to 92% in a follow-up test.

Limitations and Who Shouldn't Use This

The experiment has obvious limits. I used one free model, one free server, and only 200 samples. I couldn't control the model's sampling parameters, and the provider might have changed the model version without telling me. The time-of-day correlation could be noise, because 200 samples is a small number for seasonal analysis.

You should not use this exact approach if you need guaranteed deterministic output for things like security checks, financial calculations, or code generation that will run without review. A free model's drift can introduce subtle bugs that are far harder to catch than a simple crash.

But if you are considering a free model for a routine classification task, a retry-and-validate pattern can be surprisingly effective. Try measuring your own drift first; you might find that the model is more stable than the logs you're feeding it.

Comments

[Luis Cruz]

Your exploration of the non-determinism in free model outputs is a vital reminder of the complexities we face in automated pipelines. The fact that JSON outputs can vary, especially with numeric values and key orders, highlights the importance of robust error handling in our applications. It might be beneficial to implement a normalization step for the JSON outputs to enforce consistency before they hit your monitoring script. If you’re looking to enhance this project further, I’d be interested in discussing potential collaboration, particularly around implementing more resilient parsing strategies. What measures have you considered to mitigate these variations in future implementations?