When an agent run fails, the first suspect is the model. In most debugging sessions, the model is rarely the culprit. The harness, the context, and the budget fail first. This article is a verification playbook for two free components: free model access and a free server option. You can test both in an afternoon. The test will tell you where your failures actually live.
Consider a typical failure. A developer points at a failed refactor and says the free model is too weak. The evidence is a wrong import. The full log shows the agent received a 40,000-token context with three conflicting versions of the same file. The model picked the wrong one because the harness never specified the current version. The model was fine. The context was not.
MonkeyCode is an open-source project that offers free model access and a free server option for running agents without your own infrastructure. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The two availability claims to test are exactly those: free models and a free server. This article will not defend them. It will show you how to verify them. A claim you can test is worth more than a claim you trust.
What the Free Tier Actually Promises
Free model access means you can send prompts without a credit card. Free server means you can run the agent loop without provisioning a VM. MonkeyCode's free tier starts with 10 million tokens for model access and a free server for the agent loop. Both are useful only if they are stable enough for a real task. Stability is not a boolean. It is a distribution of outcomes. You need to sample that distribution.
The script below is a sampler. It runs a fixed agent task N times against an endpoint. It records latency, token usage, exit code, and a hash of the output. The hash lets you see whether the same input produces the same result.
A Probe That Separates Model Failure from Harness Failure
#!/usr/bin/env python3
"""probe_agent_stack.py - sample the behavior of a free model + free server.
Usage:
export AGENT_ENDPOINT="https://your-server.example/run"
export MODEL_NAME="your-free-model"
python3 probe_agent_stack.py --runs 20
"""
import argparse
import hashlib
import json
import os
import subprocess
import time
import csv
TASK = "Fix the unused import in src/main.py and run the tests."
def run_once(endpoint: str, model: str) -> dict:
payload = {
"model": model,
"task": TASK,
"max_steps": 10,
}
start = time.monotonic()
try:
result = subprocess.run(
["curl", "-s", "-X", "POST", endpoint, "-d", json.dumps(payload)],
capture_output=True,
text=True,
timeout=120,
)
wall = time.monotonic() - start
out = result.stdout.strip()
return {
"exit": result.returncode,
"wall": round(wall, 2),
"output_hash": hashlib.sha256(out.encode()).hexdigest()[:12],
"tokens": extract_tokens(out),
}
except subprocess.TimeoutExpired:
return {"exit": 124, "wall": 120.0, "output_hash": "timeout", "tokens": 0}
def extract_tokens(text: str) -> int:
try:
data = json.loads(text)
return data.get("usage", {}).get("total_tokens", 0)
except json.JSONDecodeError:
return 0
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--runs", type=int, default=20)
args = parser.parse_args()
endpoint = os.environ["AGENT_ENDPOINT"]
model = os.environ["MODEL_NAME"]
rows = []
for i in range(args.runs):
row = run_once(endpoint, model)
row["run"] = i + 1
rows.append(row)
print(f"run {row['run']}: exit={row['exit']} wall={row['wall']}s tokens={row['tokens']} hash={row['output_hash']}")
time.sleep(2)
with open("probe_results.csv", "w", newline="") as f:
writer = csv.DictWriter(f, fieldnames=["run", "exit", "wall", "tokens", "output_hash"])
writer.writeheader()
writer.writerows(rows)
if __name__ == "__main__":
main()
The script is deliberately dumb. It does not know what a good fix looks like. It only knows whether the task completed, how long it took, and whether the output changed. That is enough to separate the three failure layers.
Reading the CSV Like a Debugger
Group the rows by exit code and output hash. If all failures share the same hash, the task spec is deterministic and the harness is the problem. If failures have different hashes, the model output is unstable. If wall times grow linearly, the server is accumulating state. If wall times are flat but high, you are seeing shared-resource latency.
Here is the decision table to use after every probe run:
| Pattern | Diagnosis | Action |
|---|---|---|
| Same exit code, same hash, every run | Deterministic harness bug | Fix the task spec or the agent loop |
| Same exit code, different hashes | Model output variance | Add constraints, lower temperature, or switch free models |
| Wall time grows with run number | Server state leak | Restart the workspace between runs |
| Wall time flat but high | Shared free server | Accept latency or move to a paid tier |
| Token count doubles on retries | Context bloat | Add a budget and forbid re-reads |
The table is not a benchmark. It is a triage tool. It tells you where to look next, not what the final answer is.
A Common Probe Result
A typical probe session against a free model and a free server starts with ten clean runs. The exit code is zero every time. The output hash is identical. The wall time is flat. The verdict is boring: the stack is stable for that task.
Then you change the task to include a file that does not exist. The exit code stays zero, but the output hash changes every run. The model is inventing a plausible file path. The harness never checks whether the path exists. The free model is not the problem. The harness is missing a validation step.
That is the kind of finding the probe is built for. It converts a vague feeling about model quality into a specific, fixable gap.
What This Does Not Test
This probe does not measure model quality. It does not measure concurrency. It does not prove production readiness. Free quotas can change without notice, so run the probe weekly as a canary. If you need a contractual SLA or data residency guarantees, a free tier is not your answer.
The free server is also not a substitute for a dedicated environment. It is a place to run experiments and background jobs. Treat it as a shared resource, not a private machine. The same discipline applies to the free model tier. It is a budget, not a buffet.
The Verdict
The next time an agent fails, do not blame the model. Run the probe. Keep the CSV. The artifact will tell you whether the free model, the free server, or the harness needs the fix. That is the difference between guessing and debugging.
If you run this against MonkeyCode's free tier, share your numbers in the comments.
Top comments (0)