A Green Free Server Is Not a Repro: Six FAQ Myths

Have you ever merged because a free box went green? I have done that more than once. The job felt cheap. The result was not a repro.

Free models hide one trap. Free servers hide a second trap. People treat both like a lab. They are a noisy scratch pad. Can you replay last night's green job from git alone?

This FAQ is a myth list. Each myth is a claim I still hear. Then I show the evidence pattern. Then I give a corrected mental model.

No vendor scoreboard lives in this post. No latency leaderboard either. Those numbers rot within a week. Pin your own traces instead.

Why these myths keep spreading

Coding agents made generation cheap. Verification did not get cheaper. Teams still glue both steps together. One green check then covers two failures. That coupling is the root bug.

Want a working loop anyway? Split the work on purpose. Freeze a fixture in git. Generate in one bucket. Verify in another bucket. Record queue wait as its own number.

I keep a tiny splitter for that split. You will see it below. Treat it as an example harness. Do not treat it as a platform.

Myth 1: Free means I can skip pinning

Claim: The endpoint costs nothing, so pinning is ceremony.

Evidence pattern: Prompts drift between runs. Tool schemas drift too. Timeouts change during a late retry. The next green job is a different request. Nobody can replay it.

Corrected model: Pin the request shape, not a brand. Hash the fixture directory. Store timeout_ms next to the prompt. Keep retry_max at zero until you measure flakes.

Ask one blunt question before merge. Can a stranger replay this from git? If the answer is no, you have a demo.

git rev-parse HEAD
sha256sum fixtures/case01.tar
python3 -c "import json; json.load(open('request.json'))"

Those three commands catch most drift. They take under a minute. Skip them and the free box will lie.

Myth 2: A free server is a clean room

Claim: This box is empty, so leftover state cannot bite.

Evidence pattern: I find leftover env vars. Occupied ports break binds. Cached wheels hide missing deps. An old Docker layer saves a failing test. Green meant dirty disk.

Corrected model: A free server is shared scratch space. Prove cleanliness or assume dirt. Start from a hashed fixture every time. Write outputs into a fresh directory. Refuse inherited NODE_ENV. Refuse leftover credentials.

Would you debug flakes on five old venvs? Then do not trust a mystery box.

WORKDIR="/tmp/repro-$RANDOM"
mkdir -p "$WORKDIR"
env -i PATH="$PATH" HOME="$HOME" \
  python3 verify.py --out "$WORKDIR"

If verify needs a secret, stop. A shared box is the wrong place.

Myth 3: One green check proves generate and verify

Claim: The model wrote the patch. The same job ran tests. Green means both sides worked.

Evidence pattern: The model emitted a no-op. Tests were already green. Or it patched the assertion itself. The suite still exits zero. You just shipped a shrug.

Corrected model: Generation is only a proposal. Verification is a separate process. Different logs. Different hashes. Different exit codes. If you cannot point to both artifacts, you did not verify.

I sometimes park generate on MonkeyCode's free model access. I run verify on the free server option.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The method still works without that brand. Use any endpoint you already have. Use any box you already control. The split matters. The logo does not.

Myth 4: Retries are free, so retry until green

Claim: Tokens cost nothing tonight. Hammer the endpoint until it passes.

Evidence pattern: Retry loops hide queue storms. They also hide non-determinism. The "fix" is attempt seven. The first six attempts vanished. Your badge kept the survivor only.

Corrected model: The first run is the signal. Extra runs are a study. Cap retries at zero for the gate. Then sample flakes with a bound. Store every attempt. Never keep only the winner.

Sample rows, not a benchmark:

attempt 1  wait_ms=4100  infer_ms=1800  verify_ms=220  exit=1
attempt 2  wait_ms=900   infer_ms=1700  verify_ms=210  exit=0

That table is the story. A single green badge is not. Would you accept a test that reruns until pass?

Myth 5: Queue wait is model latency

Claim: The job took twelve seconds. Therefore the model is slow.

Evidence pattern: Ten seconds sat in a queue. One second was inference. One second was tests. People then "optimize the prompt" for a traffic jam. The prompt was never the bottleneck.

Corrected model: Split three clocks every time. Wait. Infer. Verify. Tune the bucket that actually moved. Free shared servers make queue noise worse. Queue noise is not a model bug.

Read the three clocks first. Rewrite the prompt second. Otherwise you debug the wrong layer.

Myth 6: The remote clone is my fixture

Claim: The free server has the repo, so local fixtures are extra.

Evidence pattern: The remote clone sits on a moving HEAD. Submodules never init. Golden files never land in git. A stray .env leaks into the prompt. You cannot fail the case on a train.

Corrected model: The fixture lives in git. The server only executes it. If the box vanished tonight, could you still fail locally? If not, you built a demo. You did not build a repro.

tar -C fixtures/case01 -cf - . | sha256sum
python3 verify.py --fixture fixtures/case01 --local

Local first. Remote second. Flip that order and you chase ghosts.

Artifact: a three-bucket splitter

This is example code. It is not a production harness. Swap generate() for your client. Do not commit tokens. Point MODEL_ENDPOINT at a host you already use.

#!/usr/bin/env python3
"""Three-bucket repro splitter. Example harness, not a product."""

from __future__ import annotations

import hashlib
import json
import os
import subprocess
import sys
import time
import urllib.request
from pathlib import Path


def sha_dir(path: Path) -> str:
    h = hashlib.sha256()
    for p in sorted(path.rglob("*")):
        if p.is_file():
            h.update(p.relative_to(path).as_posix().encode())
            h.update(p.read_bytes())
    return h.hexdigest()


def timed(fn):
    t0 = time.perf_counter()
    result = fn()
    ms = int((time.perf_counter() - t0) * 1000)
    return result, ms


def generate(prompt: str) -> tuple[str, int]:
    endpoint = os.environ["MODEL_ENDPOINT"]
    body = json.dumps({"prompt": prompt, "max_tokens": 256}).encode()
    req = urllib.request.Request(
        endpoint,
        data=body,
        headers={"Content-Type": "application/json"},
        method="POST",
    )

    def _call():
        with urllib.request.urlopen(req, timeout=30) as resp:
            return resp.read().decode()

    text, infer_ms = timed(_call)
    return text, infer_ms


def verify(fixture: Path, patch: str) -> tuple[int, int]:
    work = Path("/tmp") / f"repro-{os.getpid()}"
    work.mkdir(parents=True)
    (work / "patch.diff").write_text(patch)

    def _run():
        return subprocess.call(
            [
                "python3",
                str(fixture / "verify.py"),
                "--fixture",
                str(fixture),
                "--patch",
                str(work / "patch.diff"),
            ]
        )

    code, verify_ms = timed(_run)
    return code, verify_ms


def main() -> int:
    fixture = Path(sys.argv[1])
    prompt = Path(sys.argv[2]).read_text()
    fixture_hash = sha_dir(fixture)

    # Real queues should stamp enqueue vs start separately.
    wait_ms = 0
    patch, infer_ms = generate(prompt)
    code, verify_ms = verify(fixture, patch)

    record = {
        "git": subprocess.check_output(
            ["git", "rev-parse", "HEAD"], text=True
        ).strip(),
        "fixture_sha256": fixture_hash,
        "wait_ms": wait_ms,
        "infer_ms": infer_ms,
        "verify_ms": verify_ms,
        "exit": code,
        "patch_sha256": hashlib.sha256(patch.encode()).hexdigest(),
    }
    Path("repro.json").write_text(json.dumps(record, indent=2))
    print(json.dumps(record))
    return 0 if code == 0 else 1


if __name__ == "__main__":
    raise SystemExit(main())

Run it like this. Replace the host. Keep verify.py boring.

export MODEL_ENDPOINT="https://example.invalid/v1/generate"
python3 splitter.py fixtures/case01 prompts/case01.txt
cat repro.json

verify.py should apply a patch. Then it should run one test file. Nothing fancier belongs in the first loop.

Decision table

Use this after each job. Do not negotiate with it.

Observation	Trust it?	Next move
wait_ms dominates, exit 0	No	Rerun later, same fixture hash
infer_ms dominates, exit 1	Maybe	Inspect patch hash; do not retry-as-gate
verify_ms dominates, exit 1	Maybe	Rerun under env -i; check dirty disk
exit 0, missing fixture hash	No	Not a repro
exit 0, retries kept only the survivor	No	Store every attempt first
exit 0, local fixture still fails	No	Remote dirt or missing files
exit 0, local also passes, hashes match	Yes	Candidate to ship, not a trophy

A candidate can merge. A trophy is how bugs land. Which row are you actually in?

What this does not prove

This splitter does not measure quality. It does not rank models. It does not grant an SLA. Free endpoints change without a blog post. Free servers vanish without a ticket.

It also does not isolate secrets. Never send .env to a shared box. Never paste production dumps into a free model. Contract repos need a machine you control.

I am not claiming a speedup here. I am not claiming a win rate. Those claims need your traces. Mine would be stale by next week.

Who should skip this approach

Skip this if you need a compliance boundary. Skip this if tests need licensed data. Skip this if one flake pages a human. Skip this if outbound prompts are banned.

Use an isolated runner in those cases. Or keep fixtures laptop-only. Cheap is not the same as allowed.

The weekday loop I actually keep

Four steps. No dashboard. No extra badges.

1. Hash the fixture and freeze request.json.
2. Generate once and save the patch hash.
3. Verify once under env -i.
4. Write repro.json, then use the table.

If step four feels fuzzy, do not merge. Ask a sharper question instead. Which clock moved? Which hash changed? Which exit flipped?

That is the whole method. Generation is not verification. A green free server is not a repro.

If you already have a free model slot, run the splitter on one flaky job. Keep the JSON. Throw away the badge.