The scoreboard was lying, and not in a subtle way. I had a green local eval for a tiny tool-calling agent, then I pointed the same harness at a remote runner and watched “accuracy” swing like weather. The model had not gotten dumber between those two runs. The clock had moved, and I was still scoring dead sockets as wrong answers.
That is the whole claim. If you export an agent eval onto someone else’s box and collapse HTTP death into a single fail bit, you are not measuring the model. You are measuring the path. Local green, remote red, same prompt, same tool schema — and a dashboard that cannot tell the difference is just a mood ring with extra YAML.
I keep doing this to myself. A smoke eval looks honest on a laptop because the loopback interface does not take a coffee break. Then CI, or a shared runner, or a free remote box enters the story, and suddenly the agent “regressed.” Did it? Or did the handshake stall long enough for httpx to tap out at ten seconds while the model was still thinking?
The contamination is boring, which is why it wins
Agent evals already mix three different failures into one sad integer. The model can pick the wrong tool. The tool can return garbage. The transport can vanish. Only the first two belong in a quality score. The third belongs in a reliability column, and if you skip that split you will “fix” a model that was never broken.
Think of it as grading a student on whether the bus showed up. You can do that. You just should not call the result algebra.
I wanted a remote place to park the runner without standing up my own box, so I used MonkeyCode for the off-box side of this experiment. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am treating their free model access and free server option as a convenience layer — a reachable endpoint and a cheap generation path — not as a promise about hardware, quotas, permanence, or speed. The article still works if you swap in any other remote URL. The method is the point.
What I actually ran
The task is deliberately small. One user turn. One allowed tool, add_two, with two integers. A correct agent emits a well-formed tool call, the tool returns the sum, and the final message echoes that number. Anything else is a model miss. Timeouts, DNS, 5xx, empty bodies, and JSON that never showed up are transport misses. Mixing them is how pass rate becomes meteorology.
I did not chase a leaderboard. I chased a classifier I could rerun on Monday without arguing with a screenshot. The harness below is the artifact. Point LOCAL_BASE at a loopback stub and REMOTE_BASE at whatever free server or API root you actually have. Keep the prompt frozen. Keep the tool schema frozen. Change one thing: the wire.
# eval_wire.py — classify transport death vs model misses
from __future__ import annotations
import json, os, time, uuid
from dataclasses import dataclass, asdict
from typing import Any, Literal
import httpx
Outcome = Literal["pass", "model_fail", "transport_fail", "protocol_fail"]
TOOL = {
"name": "add_two",
"description": "Add two integers and return the sum.",
"parameters": {
"type": "object",
"properties": {
"a": {"type": "integer"},
"b": {"type": "integer"},
},
"required": ["a", "b"],
},
}
PROMPT = "Use add_two to compute 17 + 25. Then say only the sum."
@dataclass
class Trial:
run_id: str
target: str
outcome: Outcome
http_status: int | None
ttfb_ms: float | None
total_ms: float
error: str | None
tool_name: str | None
tool_args: dict[str, Any] | None
final_text: str | None
def classify(resp: httpx.Response | None, body: dict | None, timed_out: bool) -> Outcome:
if timed_out or resp is None:
return "transport_fail"
if resp.status_code >= 500 or resp.status_code in {408, 429}:
return "transport_fail"
if resp.status_code >= 400:
return "protocol_fail"
if not body:
return "protocol_fail"
tool = extract_tool(body)
text = extract_text(body)
if tool and tool[0] == "add_two" and tool[1] == {"a": 17, "b": 25}:
# Model did the job. Echoing the sum is extra credit, not the grade.
return "pass"
if tool or (text and text.strip()):
return "model_fail"
return "protocol_fail"
def extract_tool(body: dict) -> tuple[str, dict] | None:
# Adapt this to your chat/completions shape. Do not pretend one schema rules them all.
choices = body.get("choices") or []
if not choices:
return None
msg = choices[0].get("message") or {}
calls = msg.get("tool_calls") or []
if not calls:
return None
fn = calls[0].get("function") or {}
name = fn.get("name")
try:
args = json.loads(fn.get("arguments") or "{}")
except json.JSONDecodeError:
return None
if not name:
return None
return name, args
def extract_text(body: dict) -> str | None:
choices = body.get("choices") or []
if not choices:
return None
return (choices[0].get("message") or {}).get("content")
def one_trial(client: httpx.Client, url: str, target: str, timeout_s: float) -> Trial:
run_id = uuid.uuid4().hex[:8]
payload = {
"messages": [{"role": "user", "content": PROMPT}],
"tools": [{"type": "function", "function": TOOL}],
"tool_choice": "auto",
}
t0 = time.perf_counter()
ttfb = None
timed_out = False
resp = None
body = None
err = None
try:
with client.stream("POST", url, json=payload, timeout=timeout_s) as r:
resp = r
first = next(r.iter_bytes(), b"")
ttfb = (time.perf_counter() - t0) * 1000
raw = first + r.read()
body = json.loads(raw.decode() or "{}")
except httpx.TimeoutException as e:
timed_out = True
err = type(e).__name__
except (httpx.TransportError, json.JSONDecodeError) as e:
err = type(e).__name__
total = (time.perf_counter() - t0) * 1000
tool = extract_tool(body) if body else None
return Trial(
run_id=run_id,
target=target,
outcome=classify(resp, body, timed_out or err == "TransportError"),
http_status=None if resp is None else resp.status_code,
ttfb_ms=None if ttfb is None else round(ttfb, 1),
total_ms=round(total, 1),
error=err,
tool_name=None if tool is None else tool[0],
tool_args=None if tool is None else tool[1],
final_text=extract_text(body) if body else None,
)
def main() -> None:
local = os.environ["LOCAL_BASE"].rstrip("/") + "/v1/chat/completions"
remote = os.environ["REMOTE_BASE"].rstrip("/") + "/v1/chat/completions"
timeout_s = float(os.environ.get("EVAL_TIMEOUT_S", "12"))
n = int(os.environ.get("EVAL_N", "20"))
rows: list[Trial] = []
with httpx.Client() as client:
for target, url in (("local", local), ("remote", remote)):
for _ in range(n):
rows.append(one_trial(client, url, target, timeout_s))
print(json.dumps([asdict(r) for r in rows], indent=2))
for target in ("local", "remote"):
subset = [r for r in rows if r.target == target]
print(target, tally(subset))
def tally(rows: list[Trial]) -> dict[str, float | int]:
n = len(rows) or 1
def rate(o: Outcome) -> float:
return round(100.0 * sum(r.outcome == o for r in rows) / n, 1)
transport = [r.total_ms for r in rows if r.outcome == "transport_fail"]
return {
"n": len(rows),
"pass_pct": rate("pass"),
"model_fail_pct": rate("model_fail"),
"transport_fail_pct": rate("transport_fail"),
"protocol_fail_pct": rate("protocol_fail"),
"transport_p50_ms": None if not transport else sorted(transport)[len(transport)//2],
}
if __name__ == "__main__":
main()
Run it like a test you might actually keep, not like a demo you screenshot once:
export LOCAL_BASE=http://127.0.0.1:8080
export REMOTE_BASE=https://YOUR_REMOTE_ROOT
export EVAL_N=20
export EVAL_TIMEOUT_S=12
python eval_wire.py > wire_trials.json
Need a loopback control so “local” is not a fantasy? A stub that always emits the correct tool call is enough. You are calibrating the classifier, not auditioning a model.
# local_stub.py — control plane, not a brain
from fastapi import FastAPI
from fastapi.responses import JSONResponse
app = FastAPI()
@app.post("/v1/chat/completions")
def completions(payload: dict):
return JSONResponse({
"choices": [{
"message": {
"role": "assistant",
"content": None,
"tool_calls": [{
"id": "call_local",
"type": "function",
"function": {
"name": "add_two",
"arguments": "{\"a\": 17, \"b\": 25}",
},
}],
}
}]
})
uvicorn local_stub:app --port 8080
If local pass rate is not ~100% against that stub, stop. Your parser is the bug. I have shipped that bug. It looks like a model story in every chart you will later regret.
How I read a run without lying to myself
I refuse a single accuracy number now. Four buckets, or I do not publish the row. Pass means the tool call matched. Model fail means the bytes arrived and the agent still flinched. Transport fail means the wire blinked. Protocol fail means we could not even parse the play — wrong shape, missing choices, arguments that are not JSON.
Here is an illustrative slice, labeled because it is a fixture for the method, not a product benchmark and not a promise about any host. I am showing the shape of the argument, not a trophy.
# illustrative fixture — replace with your wire_trials.json
local {"n": 20, "pass_pct": 100.0, "model_fail_pct": 0.0,
"transport_fail_pct": 0.0, "protocol_fail_pct": 0.0}
remote {"n": 20, "pass_pct": 55.0, "model_fail_pct": 15.0,
"transport_fail_pct": 25.0, "protocol_fail_pct": 5.0}
Look at that remote line. A lazy dashboard screams 55% accuracy and someone starts swapping prompts. Split the buckets and a quarter of the pain is the wire. Another sliver is protocol drift. The model-shaped remainder is 15%, which is a different conversation than “the remote model is bad.” Same eval file. Different adult in the room.
Was the remote side slower? Sometimes. Is slower the same as wrong? Only if your timeout is a moral judgment. I keep EVAL_TIMEOUT_S in the log next to the tally because a 12-second knife and a 60-second knife do not cut the same fruit. Change the knife, change the score, and you still have not touched the weights.
One more cut that people skip: TTFB versus total time. If TTFB explodes and the body never arrives, that is not a bad tool call. That is a cold start, a queue, a proxy, a weekend. If TTFB is fine and the arguments are {"a": "seventeen"}, congratulations, you found an actual model miss. See the difference? One of those belongs in a routing ticket. The other belongs in a prompt diff.
When the free remote path is honest enough
I still use a free remote runner. I just do not let it author the grade. Smoke tests, schema checks, “does the tool name even come back,” overnight soak that I will read with the classifier on — that is fair. You get more attempts per unit of patience, and you find parser bugs faster than you find enlightenment.
I would not use it as the clock for a launch decision. I would not pour customer text into it. I would not treat a free model as a hidden gold judge and then act shocked when the judge wobbles. Cheap generation is a gift for iteration. It is a lousy supreme court.
If you need p99 promises, data residency, or a paper that pretends the network was not in the room, this approach will waste your week. Run local, or run a box you can ssh into, and keep the same classifier anyway. The classifier is the habit. The free server is just one place the habit can live.
A decision I actually use, said as sentences because a matrix makes people stop thinking. If transport_fail stays under a small slice and model_fail is stable across days, the remote path is good enough for smoke. If transport_fail moves when I change timeout and model_fail does not, I am holding a network thermometer. If protocol_fail spikes, I am on the wrong response shape and no amount of prompt poetry will help. If local stub is not a clean pass, I do not get to talk about models today.
What I will not claim
I am not going to invent a token ceiling, a GPU SKU, a durability story, or a speed ranking. Those claims go stale before the comment thread does, and I do not have a primary source in this draft that would survive a skeptic with a calendar. The reproducible part is the split: wire versus brain versus parser. You can run it on a free server, on a laptop, on a machine you regret buying. If the buckets do not exist, the percentage is theater.
So export the eval if you want. Just stop letting the socket sit for the exam. If you try the harness and the remote tally shocks you, good. That shock is the first honest number you have had all week. If you want a remote box for the runner without building one, the free server option I used is enough to get the classifier dirty — after that, the work is reading the buckets, not collecting a slogan.
Top comments (0)