If Only One Host Passed, the Agent Did Not

Your agent test is not a result yet. It is still a rumor from one host.

A second machine must run the same contract. Only then should you merge the change.

The hidden input you keep ignoring

Most agent evals pin the prompt text. Some teams even pin the tool schema.

Almost nobody pins the actual execution host. That gap is how broken loops ship.

The model is not the only moving part. Filesystem layout and network paths move too.

Clock skew quietly changes your retry behavior. DNS caches change what a tool returns.

Shell locale changes how paths expand. Home directories leak straight into tool arguments.

You currently treat those facts as noise. They are hidden inputs to every call.

Opinion: one host cannot certify an agent

Adopt a hard rule on agent evals. Fail the suite without a second host.

Local green builds hide host coupling fast. They reward agents that sniff your laptop.

Browser-only demos repeat that same mistake daily. A blessed runtime is still a single host.

The tab is not a second environment. It is one more blessed container with secrets.

If the agent needs your files, it failed. If it needs your GPU paths, it failed.

If it needs your dotenv layout, it failed. The contract has to travel without souvenirs.

Tool-calling tutorials often stop at the happy path. That is how you ship a laptop goblin.

The interesting failure is not a wrong sentence. The interesting failure is a path that only exists here.

Anatomy of a false green

Here is a common false green pattern. The local agent labels issues with flair.

It also reads your private SSH config. The remote host has no such file.

Local CI stays green and looks smart. Remote CI dies on a missing path.

That is not a flaky model problem. That is an unstated dependency on the host.

Leaked call, labeled as an example:

{
  "name": "add_label",
  "args": {
    "repo": "acme/widgets",
    "issue_id": 441,
    "label": "bug",
    "ssh_identity": "/Users/you/.ssh/id_ed25519"
  }
}

The extra label looks useful in review. The smuggled key is the entire bug.

Record a fingerprint before any model call

Do not start with graded prose output. Start with a boring host fingerprint file.

Treat that file as a test fixture. Store it beside the tool-call trace.

The script below is a labeled proposal. It is example code, not a benchmark.

#!/usr/bin/env python3
"""Proposal: fingerprint the host before an agent eval."""
import hashlib, json, os, platform, socket, sys
from datetime import datetime, timezone
from pathlib import Path

def fingerprint(cwd: str) -> dict:
    path = Path(cwd)
    return {
        "captured_at": datetime.now(timezone.utc).isoformat(),
        "hostname": socket.gethostname(),
        "platform": platform.platform(),
        "python": sys.version.split()[0],
        "cwd": str(path.resolve()),
        "cwd_exists": path.exists(),
        "user_home_in_cwd": str(Path.home()) in str(path.resolve()),
        "env_keys": sorted(
            k for k in os.environ if k.startswith(("HTTP", "NO_", "PATH", "SSH"))
        ),
        "has_local_dotenv": (path / ".env").exists(),
        "cpu_count": os.cpu_count(),
    }

def digest(doc: dict) -> str:
    blob = json.dumps(doc, sort_keys=True).encode()
    return hashlib.sha256(blob).hexdigest()[:16]

if __name__ == "__main__":
    doc = fingerprint(".")
    stable = {k: doc[k] for k in doc if k != "captured_at"}
    doc["fingerprint"] = digest(stable)
    print(json.dumps(doc, indent=2))

Run the script on your development laptop. Run the same script on a spare server.

Compare the fingerprints before you trust traces. If later calls need laptop-only keys, fail.

Allowlist the environment, then freeze it

Do not export your whole environment remotely. Export an allowlist, then freeze that list.

# Proposal: freeze env keys before the remote replay.
export AGENT_ALLOWED_ENV="PATH,LANG,TZ"
python fingerprint.py > fp.json
jq -r '.env_keys[]' fp.json > /tmp/seen_env.txt
# Fail the job if SSH_* or *_TOKEN keys appear.
if grep -E 'SSH_|_TOKEN|_SECRET' /tmp/seen_env.txt; then
  echo "env leak in fingerprint" >&2
  exit 1
fi

Extra keys are clues, not convenience features. Delete them before the second host runs.

The contract the second host must honor

Do not ship a copied chat transcript. Ship a tool-shape contract both hosts share.

The second host must see identical tools. It must return the same result shapes.

The fixture below is also a proposal. Keep it in source control with tests.

{
  "contract_id": "issue.triage.v3",
  "tools": [
    {
      "name": "list_open_issues",
      "required_args": ["repo", "limit"],
      "forbidden_args": ["token", "cookie", "ssh_identity"],
      "result_shape": ["id", "title", "state"]
    },
    {
      "name": "add_label",
      "required_args": ["repo", "issue_id", "label"],
      "forbidden_args": ["assignee_email", "ssh_identity"],
      "result_shape": ["ok", "issue_id"]
    }
  ],
  "invariants": [
    "no_absolute_local_paths",
    "no_home_directory_strings",
    "no_hostname_in_tool_args"
  ]
}

You are not grading the model's essays. You are grading the shape of calls.

Compare traces, not vibes

Write a checker that rejects host leakage. Keep the checker boring, strict, and exit-coded.

#!/usr/bin/env python3
"""Proposal: fail traces that smuggle host details into tool calls."""
import json, re, sys
from pathlib import Path

ABS_PATH = re.compile(r"(?:/Users/|/home/|[A-Za-z]:\\)")

def load(p):
    return json.loads(Path(p).read_text())

def leaks(text, fingerprint):
    hay = text.lower()
    host = str(fingerprint.get("hostname", "")).lower()
    cwd = str(fingerprint.get("cwd", "")).lower()
    if host and host in hay:
        return "hostname"
    if cwd and cwd in hay:
        return "cwd"
    if ABS_PATH.search(text):
        return "absolute_path"
    return None

def main(contract_path, trace_path, fp_path):
    contract = load(contract_path)
    trace = load(trace_path)
    fp = load(fp_path)
    errors = []
    allowed = {t["name"] for t in contract["tools"]}
    for i, call in enumerate(trace.get("tool_calls", [])):
        name = call.get("name")
        args = json.dumps(call.get("args", {}), sort_keys=True)
        if name not in allowed:
            errors.append(f"call {i}: unknown tool {name}")
            continue
        spec = next(t for t in contract["tools"] if t["name"] == name)
        missing = [k for k in spec["required_args"] if k not in call.get("args", {})]
        forbidden = [k for k in spec["forbidden_args"] if k in call.get("args", {})]
        if missing:
            errors.append(f"call {i}: missing {missing}")
        if forbidden:
            errors.append(f"call {i}: forbidden {forbidden}")
        leak = leaks(args, fp)
        if leak:
            errors.append(f"call {i}: leaked {leak}")
    if errors:
        print("FAIL")
        print("\n".join(errors))
        sys.exit(1)
    print("PASS")

if __name__ == "__main__":
    main(sys.argv[1], sys.argv[2], sys.argv[3])

Wire it into CI with plain commands. Do not hide the steps in chat.

python fingerprint.py > /tmp/fp.local.json
# Run your agent here. Write /tmp/trace.local.json.
python check_trace.py contract.json /tmp/trace.local.json /tmp/fp.local.json

Repeat every step on the second host. Diff tool shapes, and ignore extra prose.

jq -S '.tool_calls' trace.local.json > /tmp/calls.local.json
jq -S '.tool_calls' trace.remote.json > /tmp/calls.remote.json
diff -u /tmp/calls.local.json /tmp/calls.remote.json

A wording mismatch is usually just cheap noise. A shape mismatch is the actual bug.

Copy the fixtures with boring scp commands. Do not wrap them in a chat agent.

# Proposal only. Replace the remote entrypoint with yours.
scp contract.json fingerprint.py check_trace.py user@remote:/tmp/eval/
ssh user@remote 'cd /tmp/eval && python3 fingerprint.py > fp.json'

Decision table: ship, quarantine, or delete

Fill this table before you merge anything. Empty cells mean you still lack evidence.

Observation	Meaning	Action
Local pass, remote fail	Host coupling	Quarantine the agent
Both pass, shapes differ	Unstable contract	Pin tools, rerun
Both fail the same way	Honest bug	Fix the tool
Remote pass, local fail	Laptop contamination	Clean local secrets
Fingerprints identical	You did not change hosts	Use a real second machine

If you cannot complete this table, stop. You cannot honestly claim the agent passed.

Where a free remote loop actually helps

You need a second host for this. You do not need a private cluster.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open source coding project. It offers free model access and a free server option.

Use those two things as a second lane. Do not treat them as a personality test.

The free server holds the remote fingerprint. The free model access reruns the same contract.

Keep the contract file identical across both hosts. Change only the host under the runner.

This is not a performance claim at all. It is a clean separation of eval inputs.

You still must pin the tool schemas. You still must archive the raw traces.

A free lane does not replace code review. It only removes your laptop from evidence.

If you lack a spare host, use that free server. That is enough for this host-independence check.

What this does not prove

A matching remote trace is not production safety. It only shows host independence for that contract.

It does not prove the model stays stable. It does not prove the HTTP API is fast.

It does not prove the tool is authorized. It does not prove your token spend is sane.

Free access can change without any notice. Treat it as a moving worker, never a promise.

Do not publish latency numbers from one run. Do not invent model names inside eval reports.

Do not park secrets on a shared server. Redact traces before you copy them anywhere.

Shared hosts also mean you have noisy neighbors. Your eval must tolerate extra latency without cheating.

If the remote box is shared, pin timeouts. Do not loosen shapes to hide slow tools.

Who should not use this approach

Skip this pattern if you ship pure functions. Skip it if the agent exposes no tools.

Skip it if policy forbids shared remote hosts. Skip it if traces still contain customer data.

Skip it if you cannot write a contract file. Vague vibes will not survive a shape diff.

Skip it when the agent must use local GPUs. That constraint belongs on a private runner.

Do not use a free shared server for regulated data. That is the wrong class of host.

The rule to paste in the README

Put the rule in one blunt checklist. Make the merge bot enforce every line.

An agent eval is invalid until:
1. a host fingerprint is stored,
2. a tool-shape contract is stored,
3. a second host replays the contract,
4. the checker finds no host leakage.

You may argue about the exact wording. You should not argue about the underlying need.

One host remains only a polished demo. Two agreeing hosts finally count as a test.

If the second host happens to be free, fine. If the second host is yours, that is also fine.

Just stop merging rumors from a single machine.