A Tuesday deploy review stalled when a green transcript was treated as proof that an agent workflow was ready. The engineer had pointed a coding agent at a free model route and a free server, then stopped at the first successful reply. Nobody had saved the prompt hash, the response metadata, or the command that the server actually ran. The room argued about taste, while the missing record made any later comparison impossible to settle.
That scene is a composite illustration, not a customer report and not a logged personal trial. It matches a claim that shows up in review threads whenever a no-invoice path becomes available. The claim sounds practical: if the model answers and the host accepts work, the integration is validated. The corrected model is narrower, and the rest of this FAQ walks through the evidence that supports it.
Two availability claims are in scope because they change how cheap the first run feels. Operator notes for this draft state that MonkeyCode provides free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Those notes do not specify model names, token quotas, hardware, duration, or permanence, so this article will not invent them.
Is a live reply a finished trial?
Developers often repeat the claim that a successful call on a free model route closes the evaluation. The evidence against that claim is ordinary and local, because the same prompt can return different text when sampling or context shifts. A reply that compiles today can fail tomorrow if an unpinned route changes underneath the client. A finished trial needs a stored input, a stored output, and a check that another person can rerun without guessing.
Think of the reply as a weather report taped to the door, rather than as a surveyed property line. The report tells you what happened at one moment, under conditions you may not have written down. A property line is a record you can hand to someone else, with marks that do not depend on memory. Free model access makes the weather report cheap to obtain, and it does not draw the property line for you.
Is a reachable host a fixture?
A second claim treats the free server option as a private bench, simply because a shell prompt appeared. A reachable host is an access path, not a description of CPU, disk, network policy, or neighboring tenants. Without those facts, a timing number from the host cannot be compared with a number from a laptop. The corrected picture is a fixture card: what you may assume, what you measured, and what you explicitly leave unknown.
This distinction matters when a teammate tries to reproduce the task and sees a clearly different elapsed time. The difference may be load, image drift, or a different working directory, and the chat will not say which. Writing the words free server in a ticket names a procurement path, not a machine description you can rebuild. Until the card lists unknowns, the host is a place you visited, not a fixture you can hand to the next run.
Does the chat stand in for the command log?
People keep the chat transcript and drop the commands, then later cannot tell which file the server touched. A chat is a narrative of intent, while a command log is a sequence of effects with exit codes. Those two records answer different questions, and one cannot be reconstructed from the other after the session ends. The corrected habit is to store both, then judge the trial by the command log rather than by the tone of the reply.
A useful analogy is a lab notebook kept beside a photograph of the same bench after the work. The photograph shows that people were present and that glassware existed, without naming the sequence. The notebook shows which valve was turned, in which order, and what the gauge read afterward. Free model access can fill the photograph quickly, while the free server option only hosts the bench you still must log.
Can tomorrow's route replace today's pin?
Availability that is free today is still availability, and availability can change without a failing assertion in your repo. If the client never records the model identifier the API returns, a later run may silently hit a different route. The run can still look successful, because success was defined only as any non-empty reply from the route. The same risk applies to a free server whose hostname resolves, while its image, user, and working directory were never written down.
Pin what the provider returns, and label every field the provider does not return as unknown rather than as stable. Do not fill those blanks with guesses about model family, token ceiling, or machine size, because a guess becomes a false baseline. If a field is absent, the honest value is the string unknown, and the trial result must say so in the report. A pin that hides missing fields is a story about certainty, not a measurement another engineer can audit.
A bundle you can rerun
The artifact below is an unexecuted checker, not a benchmark and not a client for any particular vendor API. You obtain a response through whatever client you already use, including free model access when that is your route. You then capture commands from the host you used, including a free server when that host is the one in the notes. The checker refuses to pass when the prompt hash drifts, when exit codes are missing, or when required unknowns are blank.
It does not score prose quality, and it does not claim that two passes came from the same model build. The expected response file is whatever your client saved, as long as a text field or a content field is non-empty. Required unknown keys are a reminder of facts this draft does not have, not a list of hidden product limits. If your client returns extra fields, the report lists the keys and leaves interpretation to you.
#!/usr/bin/env python3
"""pin_trial.py — unexecuted example. Validates a trial bundle. Does not call a vendor API."""
import hashlib
import json
import sys
from pathlib import Path
REQUIRED_UNKNOWNS = (
"model_id_if_returned",
"model_version",
"token_quota",
"host_image",
"host_cpu",
"duration_or_permanence",
)
def sha256_text(text: str) -> str:
return hashlib.sha256(text.encode("utf-8")).hexdigest()
def load_unknowns(path: Path) -> dict:
data = json.loads(path.read_text(encoding="utf-8"))
missing = [key for key in REQUIRED_UNKNOWNS if key not in data]
if missing:
raise SystemExit(f"unknowns.json missing keys: {missing}")
for key, value in data.items():
if not isinstance(value, str) or not value.strip():
raise SystemExit(f"blank unknown field: {key}")
return data
def parse_commands(path: Path) -> list:
rows = []
tab = chr(9)
for line in path.read_text(encoding="utf-8").splitlines():
if not line.strip():
continue
code, sep, command = line.partition(tab)
if sep != tab or not code.lstrip("-").isdigit() or not command:
raise SystemExit(f"bad command log line: {line}")
rows.append({"exit_code": int(code), "command": command})
if not rows:
raise SystemExit("commands.log is empty")
return rows
def main(bundle: Path) -> int:
prompt = (bundle / "prompt.txt").read_text(encoding="utf-8")
fixture = json.loads((bundle / "fixture.json").read_text(encoding="utf-8"))
response = json.loads((bundle / "response.json").read_text(encoding="utf-8"))
unknowns = load_unknowns(bundle / "unknowns.json")
commands = parse_commands(bundle / "commands.log")
prompt_hash = sha256_text(prompt)
if prompt_hash != fixture["prompt_sha256"]:
raise SystemExit("prompt hash does not match fixture")
body = response.get("text") or response.get("content") or ""
if not str(body).strip():
raise SystemExit("response body is empty")
failed = [row for row in commands if row["exit_code"] != 0]
report = {
"prompt_sha256": prompt_hash,
"response_keys": sorted(response.keys()),
"model_field_present": "model" in response,
"command_count": len(commands),
"failed_commands": failed,
"unknowns": unknowns,
"pass": not failed,
}
print(json.dumps(report, indent=2, sort_keys=True))
return 0 if report["pass"] else 1
if __name__ == "__main__":
if len(sys.argv) != 2:
raise SystemExit("usage: python3 pin_trial.py ./trial-bundle")
sys.exit(main(Path(sys.argv[1])))
Create the bundle with ordinary shell commands, then run the checker against that local directory before you discuss quality. The commands below are a local workflow, and they do not open a vendor session or a remote host by themselves. Replace the sample response with a file you actually saved, or the pass only proves that the sample is self-consistent. Leave every unknown marked unknown until a provider response gives you a real value worth pinning.
mkdir -p trial-bundle
python3 - <<'PY'
import hashlib
import json
from pathlib import Path
root = Path("trial-bundle")
prompt = (
"Add a failing test for an empty command log, "
"then make the checker reject that log." + chr(10)
)
(root / "prompt.txt").write_text(prompt, encoding="utf-8")
digest = hashlib.sha256(prompt.encode("utf-8")).hexdigest()
(root / "fixture.json").write_text(
json.dumps({"prompt_sha256": digest}, indent=2) + chr(10),
encoding="utf-8",
)
(root / "response.json").write_text(
json.dumps(
{"text": "checker rejects an empty log", "model": "unknown"},
indent=2,
)
+ chr(10),
encoding="utf-8",
)
(root / "commands.log").write_text(
"0" + chr(9) + "python3 -m unittest tests.test_pin_trial" + chr(10),
encoding="utf-8",
)
(root / "unknowns.json").write_text(
json.dumps(
{
"model_id_if_returned": "unknown",
"model_version": "unknown",
"token_quota": "unknown",
"host_image": "unknown",
"host_cpu": "unknown",
"duration_or_permanence": "unknown",
},
indent=2,
)
+ chr(10),
encoding="utf-8",
)
print(digest)
PY
python3 pin_trial.py trial-bundle
Read the machine report before you argue about the prose that the model returned inside response.json. A pass means the bundle is internally consistent, not that the generated patch is correct, fast, or safe to merge. If response.json contains a model field, keep it verbatim, and still set model version to unknown unless a version was actually returned. If you used the free server option, paste real exit codes, and never invent hardware fields to look complete.
A second command shows the failure mode the fixture is meant to catch before anyone calls the run a trial. An empty log, or a log without exit codes, should stop the checker with a clear error rather than a quiet pass. Run that negative case on purpose, because a checker that has never failed is an untested checker. Keep the failing log out of the passing bundle so the two results stay easy to tell apart.
python3 - <<'PY'
from pathlib import Path
Path("trial-bundle/commands.log").write_text("", encoding="utf-8")
PY
python3 pin_trial.py trial-bundle
What this does not show
The checker cannot prove isolation, uptime, or a token ceiling, because those facts are absent unless a provider actually returned them. It cannot compare two model families, because this draft has no verified model names and will not borrow names from a marketing page. It also cannot turn one green pass into a load test, a security review, or a forecast of what the route will cost later. Treat a pass as a gate on record-keeping, then add separate checks for tests, diff review, and secret scanning before a merge.
You should not use this approach for production traffic, customer data, or any host whose tenancy and retention you have not read. You should not use it if you need a service-level agreement, a published quota, or a hardware baseline, since none were supplied here. You should not use it as a ranking method across tools, because a single fixture has no statistical power worth citing. Teams that cannot store prompts and logs should not start here, because the method is the record, not the route that happened to answer.
Confirm current terms before you depend on free model access or the free server option, since a label in a draft is not a contract. If those two options are how you capture the bundle, use them only to fill the response file and the command log. Keep every unstated limit visible in the report, and treat that marked unknown as part of the result. The practical next step is to pin one small trial this way, and to leave every limit the provider did not state marked unknown.
Top comments (0)