How I Test Free AI Model Consistency in 10 Prompts

A free AI model is stable enough to use when the same prompt, repeated ten times, produces a clear majority answer and an average pairwise similarity above 0.8. I measure that with exact-match rate, difflib similarity, and a meaning classifier before I put any free model in front of users.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Why I run a consistency test before I ship

I ran the same prompt ten times. I got ten different answers. Three were right. Seven were wrong. That was my first week with a free model.

I almost shipped it. I had a chatbot that answered customer questions. The first answer looked great. The second was fine. The third invented a refund policy that did not exist. That is when I stopped.

A consistency test is not about finding the "best" answer. It is about measuring how much a model's output varies for the same input. Hallucinations are one symptom of instability. If a model invents a policy once, I need to know how often that happens.

Consistency is not accuracy. A model can be consistently wrong. But consistency tells me when to add a human check. A single demo hid the invented policy; ten repeats did not.

What this test gives me that a one-shot demo does not:

• A repeatable sample of ten answers for one prompt
• An exact-match rate so I can see if the model locks onto one string
• An average similarity so I can catch reworded but related answers
• A meaning-level label so I can separate correct, partial, and wrong

Compared with a polished demo, ten repeats are slower and less flattering. That is the point. One good answer is a sample of one. Ten answers are a distribution I can score.

How I run the 10-prompt consistency test

I use a small Python client against a chat-completions endpoint. Nothing in this setup is a production SLA. It is a lab loop I can rerun after I change a prompt.

What I use:

• Python 3.10 or newer
• The requests library (docs)
• A MonkeyCode free server endpoint and a free model name (I check my account for the current list)

I set up the environment:

export MONKEYCODE_FREE_SERVER='https://your-endpoint.example/v1'
export MONKEYCODE_MODEL='your-free-model'

I verify the setup:

python -c "import requests; print('ok')"

I should see ok. If not, I install requests with pip install requests.

Step 1: Call the same prompt ten times

I write a client that calls the model, then I call it in a loop:

import os
import requests

SERVER = os.environ['MONKEYCODE_FREE_SERVER']
MODEL = os.environ['MONKEYCODE_MODEL']

def ask(prompt: str) -> str:
    resp = requests.post(
        f'{SERVER}/chat/completions',
        json={
            'model': MODEL,
            'messages': [{'role': 'user', 'content': prompt}],
        },
        timeout=30,
    )
    resp.raise_for_status()
    return resp.json()['choices'][0]['message']['content']

prompt = 'Write a one-sentence summary of a refund policy.'
answers = [ask(prompt) for _ in range(10)]
for i, answer in enumerate(answers):
    print(f'{i}: {answer}')

I run it:

python consistency.py

I will see ten answers. Some will match. Some will not. That is the data. I do not judge it yet. I just collect it.

I keep the prompt fixed on purpose. If I rewrite the prompt between calls, I am no longer measuring the model. I am measuring my own edits. For this refund example, I want to know whether the same sentence request stays on policy or drifts into invention.

How I measure consistency: exact match vs similarity

I always compute two scores. Exact match is strict. Similarity is looser. Neither one is accuracy. Together they tell me whether the model is a fixed string, a cluster of rewrites, or a moving target.

Exact-match rate

I count how many answers are identical:

from collections import Counter

counts = Counter(answers)
most_common = counts.most_common(1)[0]
print(f'Most common answer: {most_common[1]}/10 times')

A high number means the model is stable on the string. A low number means it is not. Exact match is not enough on its own. Two answers can differ in wording and still be correct. A single extra comma drops an otherwise identical sentence out of the exact-match bucket.

Average pairwise similarity

I use a simple similarity check with Python's difflib (documentation):

import difflib

def similarity(a: str, b: str) -> float:
    return difflib.SequenceMatcher(None, a, b).ratio()

pairs = [(answers[i], answers[j]) for i in range(10) for j in range(i + 1, 10)]
avg_sim = sum(similarity(a, b) for a, b in pairs) / len(pairs)
print(f'Average similarity: {avg_sim:.2f}')

Score	Meaning
Above 0.8	Answers are close in wording
0.6 – 0.8	Some variation, still related
Below 0.6	Answers diverge significantly

I use these thresholds as a starting point. A different task may need different ones. Exact match breaks on tiny punctuation changes. Similarity catches rewrites. A model that says the same thing every time is predictable, and predictable is reviewable. A model that says something new each time is a moving target.

If exact-match is low but average similarity is above 0.8, I treat that as stable wording with light paraphrase. If both are low, I do not build a user-facing path on that prompt.

How I classify failures and decide whether to trust the model

Similarity scores hide the interesting part. I read the answers and group them by meaning:

def classify(answer: str) -> str:
    if 'refund' in answer.lower() and '30' in answer:
        return 'correct'
    if 'refund' in answer.lower():
        return 'partial'
    return 'wrong'

labels = [classify(a) for a in answers]
print(Counter(labels))

Classification criteria I used for this prompt:

• correct: mentions refund and the 30-day window
• partial: mentions refund but misses the window
• wrong: does not mention refund at all

Now I see the real picture. Maybe five are correct, three are partial, and two are wrong. That is the evidence I need — not a single lucky demo. The classifier above is a keyword stub for this refund prompt, not a general judge. I change the rules when the task changes.

Decision rules I actually use:

• Exact-match rate below 30% — the model is too unstable for the task. I do not build on it.
• Average similarity above 0.8 — I use the model with a verification step.
• Classification shows a clear winner — I route that answer to the user and log the rest.

A model that is 50 percent consistent is a coin flip. A model that is 90 percent consistent is usable with a review step. Classification tells me which parts of the answer are stable and which are not.

After I ran this test, I changed the design. Instead of showing the model's answer directly, I showed three options and asked the user to pick. The consistency problem became a user-choice problem. That is the design shift the numbers gave me.

Limitations and what I do next

This test measures one model on one prompt. It does not measure all tasks. A model can be consistent on summaries and inconsistent on code. I run this test for each task I care about. I also rerun it after I change the prompt, because a new sentence is a new experiment.

Free servers have variable latency. I do not use this for a production SLA. I do not use it for medical or legal decisions. The test is a tool, not a certification.

Who should skip this:

• Teams with strict accuracy requirements
• Teams that need certified model behavior
• Anyone who expects a free model to be deterministic

Free models are probabilistic by design. The test helps me understand the probability, not remove it. Ten answers. One prompt. That is the whole experiment. I run it before I build. I run it again after I change the prompt. The numbers tell me when something shifted.

A free model is a tool, not a promise. Measure it before you trust it. If you want to try this flow, MonkeyCode's free server and a free model are a fine place to start. Export the two environment variables, paste the loop, and score ten answers before you wire the model to users — the run takes less than five minutes.