Learn Why Identical Prompts Return Different Outputs by Building a Tiny Repeatability Probe

Identical prompts do not imply identical model behavior. I built a tiny probe that sends one request three times and measures text similarity, token counts, and latency. This is the kind of output I look for; your free endpoint will differ.

factual
latency_s [0.412, 0.398, 0.405] median 0.405
same_text True
pairwise_similarity [1.0, 1.0]
completion_tokens [3, 3, 3]
----------------------------------------
opinion
latency_s [0.501, 0.522, 0.515] median 0.515
same_text False
pairwise_similarity [0.72, 0.68]
completion_tokens [28, 31, 27]

The single learning question: can I separate harmless text variation from a consistency problem that would break a downstream script?

Recent AI news has debated whether model text is detectable or watermarked. I don't have a primary-source answer for that, so I stayed with a property I can test without trusting a headline: repeatability. I used MonkeyCode's advertised free model access as the lab.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The operator describes MonkeyCode as an open-source project with free model access, an advertised 30M token tier, and a free server option. I treat the 30M token number as a dashboard value to check, not a permanent guarantee, because free quotas and routes change often.

What you need

• Python 3.12+
• requests 2.32+
• One terminal or notebook
• Base URL, API key, and model name from your current dashboard
• Do not paste real secrets into a shared notebook

For request behavior, see the requests docs. For median, see the statistics docs.

What the probe does

1. Send three identical requests at temperature=0.0.
2. Compare normalized text with SequenceMatcher.
3. Record median latency and completion token count.
4. Use a factual prompt as the control and an open-ended prompt as the variation test.

Run the probe

import os
import time
import requests
from statistics import median
from difflib import SequenceMatcher

BASE = os.environ.get('MONKEYCODE_BASE_URL', 'http://localhost:8000/v1')
KEY = os.environ.get('MONKEYCODE_API_KEY', '')
MODEL = os.environ.get('MONKEYCODE_MODEL', 'replace-me')

def call(prompt, temperature=0.0, max_tokens=64):
    payload = {
        'model': MODEL,
        'messages': [{'role': 'user', 'content': prompt}],
        'temperature': temperature,
        'max_tokens': max_tokens,
    }
    t0 = time.perf_counter()
    r = requests.post(
        f'{BASE}/chat/completions',
        json=payload,
        headers={'Authorization': f'Bearer {KEY}'},
        timeout=30,
    )
    elapsed = time.perf_counter() - t0
    r.raise_for_status()
    data = r.json()
    text = data['choices'][0]['message']['content']
    usage = data.get('usage', {})
    return {
        'text': text,
        'seconds': round(elapsed, 3),
        'prompt_tokens': usage.get('prompt_tokens'),
        'completion_tokens': usage.get('completion_tokens'),
    }

def similarity(a, b):
    return round(SequenceMatcher(None, a.lower(), b.lower()).ratio(), 3)

prompts = {
    'factual': 'What is the capital of Nova Scotia? Answer in one word.',
    'opinion': 'Write one sentence about why students should learn ML.',
}

for name, prompt in prompts.items():
    results = [call(prompt, temperature=0.0) for _ in range(3)]
    latencies = [r['seconds'] for r in results]
    texts = [r['text'].strip() for r in results]
    print(name)
    print('latency_s', latencies, 'median', median(latencies))
    print('same_text', len(set(texts)) == 1)
    print('pairwise_similarity', [similarity(texts[0], t) for t in texts[1:]])
    print('completion_tokens', [r['completion_tokens'] for r in results])
    print('-' * 40)

Replace MONKEYCODE_BASE_URL, MONKEYCODE_API_KEY, and MONKEYCODE_MODEL with the current values from your dashboard. The path /chat/completions may not match every provider; check the current endpoint documentation rather than assuming it.

How to read the output

• factual with same_text True and a one-word answer means low variance for a tightly constrained prompt.
• opinion with same_text False means free endpoints can still vary even at temperature=0.0. Treat every output as non-deterministic unless the provider pins a seed.
• completion_tokens varying between 3 and 28 is expected for different prompt types. If token counts change wildly for the same prompt, inspect max_tokens, stop sequences, or server-side caching.

This output block is the kind of result I look for, not a benchmark result. Run the probe on your own endpoint to get your own numbers.

One error input

Send an invalid max_tokens: a negative value should not be silently accepted.

try:
    call('What is the capital of Nova Scotia? Answer in one word.', max_tokens=-1)
except requests.HTTPError as e:
    print(e.response.status_code, e.response.text[:120])

If the API returns 400 with a clear validation message, the server is doing basic input checking. If it returns 500 or silently truncates the output, treat that as a red flag before building prompts around max_tokens.

Common mistakes

• Forgetting to set environment variables. The probe fails with 401 or a missing key message.
• Hardcoding a real API key in the file. Use environment variables.
• Using a timeout that is too low. A slow free server can look like a deterministic failure when it is just throttling.
• Treating one endpoint's output as a measure of model quality. This probe checks repeatability, not correctness, safety, or relevance.

Limitations

• One free endpoint is a snapshot. Model names, route paths, and token caps can rotate.
• This probe does not test factual correctness. A model can consistently return the wrong answer.
• A free server can be slow or rate-limited. Use it as a lab, not as a production dependency.
• The 30M token and free server claims are operator-supplied. Check the current quota before you build a pipeline on top of them.

Who should not use this approach

• Anyone who needs stable latency, HIPAA compliance, production uptime, or guaranteed determinism.
• Anyone whose downstream code assumes identical JSON fields between calls.
• Anyone who wants to evaluate model quality with one run. Three calls are weak evidence; use 10 to 20 calls for a better signal.

What you should understand after running it

• LLM APIs have nondeterminism. The same prompt can produce different text, token counts, and latency.
• Token usage comes from the usage object, not from len(text).
• A factual single-word prompt is a useful control fixture. An open-ended prompt exposes variation.
• Median latency from three calls is only a rough signal, not a report.

Extension exercise

• Add a temperature sweep of [0.0, 0.5, 1.0], run each 10 times, and compare pairwise similarity across the set.
• Guess before running which fixture will fail. The factual prompt may still drift on a less constrained model.
• Corrupt one response fixture by deleting the usage key and observe where the probe breaks.

If a free server is already on your screen, run the probe with three prompts before you copy any model advice into your notes. Better to measure a small failure now than debug a silent one later.