On a Friday afternoon a backend team parked a flaky checkout test on a courtesy box that nobody billed to a cost center. The agent looped, the test went green, and someone pasted a victory screenshot into the release channel before dinner. Monday morning the same prompt and repository on that free endpoint produced a timeout, a different patch, and a reverted merge. The screenshot was not a lie, but it was also not a measurement that another engineer could rebuild.
Stories like that now travel faster than lab notes, especially while public threads ask whether models already outcode most working developers. A courtesy slot feels like a laboratory because the prompt matches and the git SHA is visible in the header. It is closer to a practice room with a hallway door that never quite latches during rush hour. This FAQ treats slogans around free model access and free servers as claims, then replaces them with a protocol you can run.
Claim: two agents on a free model are already a fair fight
Developers repeat this because the invoice is empty and the model label in the UI looks identical across tools. An empty invoice does not pin temperature, routing, queue depth, or a system preamble that can rotate without a public changelog. Two coding agents that both used a free model can still differ in tool schemas, retry policy, and truncated function-call handling. If you cannot serialize those choices beside the model label, you compared product surfaces rather than a controlled independent variable.
A corrected mental model is simple and slightly unromantic, which is usually a good sign in evaluation work. Name the independent variables you actually control, and treat everything else as weather that must be logged. Weather can be recorded in a manifest that sits next to the patch and the failing test command. Weather cannot be wished into a constant because a dashboard badge happened to say free.
Claim: a free server is quiet enough to time an agent loop
This claim arrives with a stopwatch screenshot and a proud median that looks ready for a conference slide. Shared machines have noisy neighbors, lukewarm caches, and disks that still remember the last tenant's node_modules directory. Timing an agent on courtesy capacity is like timing a sprinter on a sidewalk that also hosts a food cart. You may still learn whether the runner finishes, which is useful triage and a poor ranking statistic.
If you need duration at all, record load averages, IO wait, and whether the working tree was already warm. Then label the number as a rehearsal observation rather than as a benchmark you would defend in public. A rehearsal observation can still kill a bad idea before it reaches a pull request template. It cannot honestly rank two vendors in a blog table that pretends the sidewalk was a track.
Claim: extra loops are free science because the invoice is empty
Token invoices can be zero while selection bias remains expensive in ways that never appear on a bill. Each extra loop on the same failing test is a peek at the answer key, even when nobody reads the patch. The agent that eventually greens the suite after fourteen attempts is not the agent you will get in one-shot review. Courtesy capacity makes that peek feel harmless because nobody's card is charged for the extra attempts.
The scientific cost is still paid in overfit patches and stories that cannot be rerun under a freeze. Treat a free loop budget as a rehearsal allowance, not as permission to fish for a prettier transcript. Decide the attempt cap before the first model call, and write that cap into the trial manifest. When the cap is hit, record failure and stop, because fishing is debugging rather than evaluation.
A rehearsal room is still worth booking when the alternative is waiting for a dedicated runner that finance has not approved. MonkeyCode is relevant here only as one coding-agent workflow that currently offers free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. That class of courtesy capacity is exactly what this FAQ is trying to keep from being mistaken for a sealed lab.
Claim: the transcript is the experiment
Teams export a chat log and believe they have captured causality because the words on screen look complete. The log rarely includes the runner fingerprint, the dirty files that were never committed, or the network path that served the model. Two transcripts can rhyme in tone while the worktrees diverge in lockfiles, generated clients, and ignored build artifacts. If you cannot rebuild the working directory and the tool boundary, you have literature rather than an experiment.
Claim: green on a courtesy slot predicts green in CI
CI is a different social contract from a courtesy box, even when both machines happen to run the same distribution. Pipelines usually pin images, restrict network, and refuse to honor a laptop's cached credentials or interactive prompts. Courtesy boxes are hospitable by design, while CI is paid to be suspicious of everything the patch forgot to declare. A patch that needs hospitality will fail the moment it meets suspicion, which is a property of the patch.
You should map the hospitality you accidentally relied on, including outbound package fetches and ambient environment variables that CI will not provide. Then decide whether the patch is a gift to the repository or a liability that only survives in a warm room. Hospitality is not a character flaw in free servers. It is a missing constraint that your protocol has to add back by hand.
A recorder that refuses to pretend
The artifact below is a trial recorder rather than a leaderboard, and it is intentionally boring on purpose. It writes a fingerprint of the runner, a freeze of the git tree, and a cap on attempts before any model is called. Copy the files into a throwaway directory, then run them against a failing test you already understand. If the recorder cannot complete, you do not have an evaluation yet; you have a setup bug to fix first.
fingerprint.sh captures the room.
#!/usr/bin/env bash
# fingerprint.sh — record the room, not the myth
set -euo pipefail
export TRIAL_OUT="${1:-./trial_fingerprint.json}"
python3 <<'PY'
import json, os, platform, socket, subprocess, time
from pathlib import Path
def sh(cmd: str) -> str:
try:
p = subprocess.run(cmd, shell=True, capture_output=True, text=True, timeout=20)
return (p.stdout or "").strip()
except Exception as exc:
return f"error:{type(exc).__name__}"
out = Path(os.environ["TRIAL_OUT"])
payload = {
"captured_at_unix": int(time.time()),
"hostname": socket.gethostname(),
"platform": platform.platform(),
"python": platform.python_version(),
"cwd": os.getcwd(),
"loadavg": list(os.getloadavg()) if hasattr(os, "getloadavg") else None,
"git_head": sh("git rev-parse HEAD"),
"git_branch": sh("git branch --show-current"),
"git_status_short": sh("git status --porcelain"),
"disk_cwd": sh("df -k . | tail -n 1"),
"opaque_model_label": os.environ.get("MODEL_LABEL", ""),
"env_names": sorted(
k for k in os.environ
if k.startswith(("CI", "RUNNER", "NODE", "VIRTUAL_ENV", "GOPATH"))
),
}
out.write_text(json.dumps(payload, indent=2) + "\n")
print(f"wrote {out}")
PY
record_trial.py freezes the protocol before the agent is allowed to touch the tree.
#!/usr/bin/env python3
"""record_trial.py — freeze the protocol before the agent runs."""
from __future__ import annotations
import argparse
import json
import subprocess
import sys
import time
from pathlib import Path
def run(cmd: list[str], timeout: int) -> dict:
started = time.time()
proc = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout)
return {
"cmd": cmd,
"returncode": proc.returncode,
"seconds": round(time.time() - started, 3),
"stdout_tail": (proc.stdout or "")[-2000:],
"stderr_tail": (proc.stderr or "")[-2000:],
}
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("--test", required=True, help="failing test command")
parser.add_argument("--attempt-cap", type=int, default=2)
parser.add_argument("--timeout", type=int, default=120)
parser.add_argument("--out", default="trial_manifest.json")
parser.add_argument("--fingerprint", default="trial_fingerprint.json")
args = parser.parse_args()
fingerprint = Path(args.fingerprint)
if not fingerprint.exists():
print("run ./fingerprint.sh first", file=sys.stderr)
return 2
test_cmd = args.test.split()
baseline = run(test_cmd, args.timeout)
manifest = {
"role": "rehearsal_not_lab",
"attempt_cap": args.attempt_cap,
"attempts_used": 0,
"baseline_test": baseline,
"fingerprint_file": str(fingerprint),
"publishable_as_ranking": False,
"rule": "Do not raise attempt_cap after seeing a near-miss patch.",
}
Path(args.out).write_text(json.dumps(manifest, indent=2) + "\n")
print(f"baseline returncode={baseline['returncode']} wrote {args.out}")
if baseline["returncode"] == 0:
print("refusing to start: the test is already green, so there is no trial")
return 1
print("Call your agent only after this freeze. Stop at attempt_cap.")
return 0
if __name__ == "__main__":
raise SystemExit(main())
A first rehearsal looks like the following, including the unglamorous refusal when the suite is already green.
chmod +x fingerprint.sh record_trial.py
export MODEL_LABEL="free-endpoint-label-from-config-not-memory"
./fingerprint.sh
python3 record_trial.py --test "pytest -q tests/test_checkout.py" --attempt-cap 2
git add trial_fingerprint.json trial_manifest.json
git diff --cached --stat
After the freeze, every agent attempt should increment attempts_used in that JSON file. A reviewer should refuse to edit attempt_cap upward after seeing a near miss. If the agent proposes a patch, keep the recorder files in the same commit as the production change. Reject the controlled-run claim whenever those recorder files are missing from the reviewable diff. The point is not ceremony for its own sake. The point is to make it socially awkward to promote a sidewalk sprint into a stadium result.
What the numbers are allowed to mean
| Observation | Treat as | Do not treat as |
|---|---|---|
| Rehearsal green on a courtesy slot | a reason to open a draft pull request | a vendor ranking |
| Timeout while loadavg is elevated | weather on a shared sidewalk | proof the agent is slow |
| Success after loops past the cap | a debugging anecdote | a reported pass |
| Transcript without a fingerprint | literature | an experiment |
| Green here, red in CI | missing constraints | a flaky model |
Read the table as a social contract rather than as a scoring rubric. Courtesy capacity is good at answering a modest question: can this loop even produce a plausible patch on a real failing test. It is bad at answering a louder question: which system should a team adopt, fund, or quote in a hiring loop. If you need the louder answer, move the same recorder onto a pinned runner that you actually own.
Limitations, and who should not use this
This protocol does not convert a shared queue into a laboratory, and it does not attest that the fingerprint is tamper proof. Load averages are coarse, df is not a storage trace, and an opaque model label is only as honest as the configuration you copied. Courtesy access can change shape without a blog post, so a result from last week is not a result from today even when the SHA matches. The scripts also assume a Unix-like shell, a git worktree, and a test command that fails for a reason you already understand.
Do not use this approach if you are publishing a league table, forecasting production latency, or briefing a promotion packet with a single green screenshot. Do not use it if the repository cannot be tested without secrets that a courtesy server should never hold. Do not use it as a cost model, because an empty invoice is not a unit of scientific validity. In those cases, rent or reserve an isolated runner, pin the image, and keep the same recorder so the notebook format stays familiar.
The corrected mental model is small enough to write on a sticky note. A courtesy slot is a practice room, and practice rooms are valuable when you refuse to sell the recital from the hallway. Record the weather, cap the peeks, and keep hospitality visible. If the run still looks good after that, you have a draft worth taking to CI, which is the first place a patch has to live without applause.
Top comments (0)