Myth FAQ: A Green Demo Is Not a Passing Test

You ran the prompt once. It returned something sensible. You screenshotted it and moved on.

Then a teammate ran the same prompt. It returned an empty list, or a wall of apology text, or nothing at all.

Was the model broken? Probably not. Your evidence was.

This is a myth-busting FAQ for anyone prototyping against a free model endpoint. Each myth gets a cheap check you can run, plus a corrected mental model. The artifact is a small harness you paste into your repo. It writes evidence to JSON instead of screenshots to Slack.

One note on scope: I'm pointing this at MonkeyCode's free model access and the free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am not quoting model names, quotas, hardware, or benchmarks here, because I have not verified any of them. Check the current terms yourself before you build a plan on top of them.

Myth 1: "It worked twice, so it works."

Two clean runs feel like proof. They are not proof. They are a coin flip with extra steps.

A prompt can pass because of a warm cache, a short input, or plain luck. Run the same input five times and log what changes between runs. That diff is your first real evidence.

What to record per run:

• exit status of your own code, not just the remote call
• transport outcome, including the exception type when it fails
• retry count for that run
• machine-checked output validity, not readability
• wall-clock seconds measured in your process

If any of those columns is empty, you have a demo. You do not have a test.

Corrected model: reliability is a distribution you sample, not a feeling you have after one nice answer.

Myth 2: "The output looked right."

Text that reads well is not evidence. Your eyes are a terrible schema validator.

Pick a contract a machine can check in under a millisecond. For a code-generation prompt, that might be:

1. Does the response parse as JSON?
2. Does it contain a steps array?
3. Is every step a non-empty string?
4. Does the first step reference a path that actually exists?

If a cheap assertion can answer the question, stop asking a human to answer it. Humans are slow, and they get bored around run three.

Corrected model: plausible output and correct output are different properties. Only one of them is testable.

Myth 3: "Free access means cost is zero."

Cost does not disappear when the invoice does. It changes units.

You stop paying dollars per call. You start paying wall-clock time, retry loops, and your own attention. Those are real costs, and they are harder to see on a dashboard.

This is where free access genuinely matters for the workflow below. MonkeyCode's free model access and free server option are the two claims I'd point you at if you want to run this harness end to end without first provisioning your own endpoint and paying for it. Those are operator-supplied availability claims, not measured results.

Budget in the right unit:

• If your run takes 40 seconds, a hundred runs is an afternoon, not a coffee break.
• If one in five runs retries, your real request volume is 25 percent higher than your loop count.
• If you never time the run, you cannot tell a slow model from a slow script.

Corrected model: a free endpoint lowers money cost. It does not lower the cost of being wrong.

Myth 4: "A successful call means the task succeeded."

Your HTTP client's success is not your task's success. They are different events.

A transport-level success tells you bytes arrived. It says nothing about whether those bytes satisfy your contract. Common ways this bites:

• The client returns a body that parses, but the shape is wrong.
• A retry fires after a partial write, so you keep two half-results.
• Your timeout is shorter than a legitimate cold path, so you cancel good work.
• Your loop swallows an exception and reports an empty string as a pass.

Every one of those failure modes lives in your code, not in the model. Instrument the seam between them.

Corrected model: two success signals exist. Log them separately, or you will merge a transport pass with a task fail.

Myth 5: "When it fails, the model is at fault."

Blame is the last step of triage, not the first. Work outward from your own process.

Symptom	Likely layer	First check
Fails before any network activity	harness	imports, env vars, entrypoint path
Timeout or connection error	transport	timeout value, payload size, retry policy
Valid JSON, wrong shape	prompt contract	the schema you stated in the prompt
Valid shape, wrong content	model capability	narrow the task, add a checked example
Passes locally, fails deployed	environment	env vars, working directory, relative paths

Read the table top to bottom. Most teams jump straight to row four and skip rows one and two entirely.

Corrected model: classify the failure layer before you change the prompt.

The harness

This is a runnable template, not a report. Wire call_model to your own client and run it in your environment. It has no third-party dependencies, so it drops into any Python 3 project.

# verify_once.py - run N times, assert structure, keep evidence.
import json
import time
from dataclasses import dataclass, asdict

RUNS = 5
MAX_RETRIES = 2


@dataclass
class Run:
    i: int
    ok: bool
    seconds: float
    retries: int
    reason: str = ""


def call_model(prompt: str) -> str:
    """Wire this to your own client. Return raw text. Raise on transport failure."""
    raise NotImplementedError


def assert_shape(text: str):
    """Machine-checkable contract. No vibes allowed."""
    try:
        obj = json.loads(text)
    except json.JSONDecodeError as e:
        return False, f"not json: {e.msg}"
    if not isinstance(obj.get("steps"), list):
        return False, "missing steps[]"
    if not obj["steps"]:
        return False, "empty steps[]"
    for s in obj["steps"]:
        if not isinstance(s.get("cmd"), str) or not s["cmd"].strip():
            return False, "step without cmd"
    return True, ""


def main():
    prompt = open("prompt.txt", encoding="utf-8").read()
    runs = []
    for i in range(1, RUNS + 1):
        t0 = time.perf_counter()
        retries, text, reason = 0, "", ""
        while retries <= MAX_RETRIES:
            try:
                text = call_model(prompt)
                break
            except Exception as e:  # timeout, transport, rate limit
                retries += 1
                reason = f"transport: {type(e).__name__}"
                time.sleep(2 ** retries)
        ok, why = assert_shape(text) if text else (False, reason or "no output")
        runs.append(Run(i, ok, round(time.perf_counter() - t0, 2), retries, why))

    wall = [r.seconds for r in runs]
    print(f"passed {sum(r.ok for r in runs)}/{RUNS}")
    print(f"wall min={min(wall):.2f}s max={max(wall):.2f}s "
          f"spread={max(wall) - min(wall):.2f}s")
    print(f"total retries={sum(r.retries for r in runs)}")
    for r in runs:
        print(f"  run {r.i}: ok={r.ok} {r.seconds}s retries={r.retries} {r.reason}")
    with open("evidence.json", "w", encoding="utf-8") as f:
        json.dump([asdict(r) for r in runs], f, indent=2)


if __name__ == "__main__":
    main()

Commit evidence.json next to the prompt. Now your claim travels with a file instead of a memory.

How to read the output

• passed 5/5 on five runs is a smoke test. It is not a reliability number.
• One failure in five means the contract is not settled yet. Fix the contract first.
• Wall-clock spread matters more than the mean when you retry.
• Retries inflate real request volume. Track them, or they hide the original failure.

What this harness does not prove

Be honest about the ceiling here.

• Five runs is a smoke test. Statistical confidence needs far more samples and a stated method.
• It checks shape, not correctness. A well-formed wrong answer still passes.
• It measures your machine's wall clock, including your own overhead. That is the number you actually feel, but it is not a server-side latency claim.
• It says nothing about limits, quotas, or availability windows. Those change, and you should read them at the source.

When a free endpoint fits, and when it does not

Situation	Use free access?	Reason
One-off script or internal tool	Yes	Failure cost is low, and you learn the failure modes
Building your first assertion harness	Yes	Cheap iterations while the contract is unstable
Prompt contract still changing daily	Yes	You will throw the code away twice anyway
Deadline-critical production path	No	You need a support path and a stated capacity
Data you cannot send to a third party	Only after reading the terms	Consent is a legal question, not a technical one
Large batch jobs	Measure first	Wall-clock cost is the real bill

Who should not use this approach

• Teams that need a contractual latency number or a named support channel.
• Anyone handling regulated data who has not read the current terms.
• Anyone who has not written the assertions yet. Build the check before the volume.
• Teams expecting a demo to count as a test. This workflow will only disappoint you.

The corrected mental model

Three lines to carry forward.

1. A demo is one sample. A test is a recorded distribution.
2. Readability is not validation. Assertions are.
3. Free changes the price, not the proof.

Run the harness five times before you write the design doc. If you want to try it against a free endpoint, MonkeyCode's free model access is a reasonable place to start. Bring your own assertions.