48-Hour Field Notes: The Remote Job Exited Clean. The Score File Was Still the Fixture.

I spent the first evening sure the remote job had called a live model, because the process exited cleanly and the log said completed. The score file matched my laptop, including a failing case I had planted on purpose before the upload. Have you ever trusted an exit code more than the artifact that exit code claimed to produce? I had trusted that same signal on quieter nights, and this run made the habit look expensive.

The job was a tiny eval harness, not a product demo, and I only needed one honest completion per prompt case. I wanted a place to run that harness away from my laptop without renting a long-lived machine for a short check. MonkeyCode's free model access and free server option were the two availability claims I was willing to try here. Disclosure: This article was prepared as part of MonkeyCode's product outreach, so treat product mentions as outreach context rather than a measured bake-off.

What I thought I was testing

I was not trying to rank models, and I was not trying to publish a latency number I could not stand behind. I was trying to answer a narrower question about whether this remote process actually left the fixture branch. If the answer is no, every later opinion about prompt quality is just a story about my stub. Would you grade a live service with a file you committed last week and then forgot to replace?

The harness reads a small JSONL file of prompts, calls one client function, and writes a score file beside the run directory. Each row stores a case id, a source label, a run id, and a short normalized answer for later review. I kept the client boring on purpose, because clever clients hide the exact branch that lied to me last night. The fixture path exists so local unit tests can run without a network, which is also how this bug became comfortable.

What I tried across the two days

On the first night I copied the repo, created a virtual environment, and ran the harness with a flag I believed meant live. The remote process printed a friendly completed line, exited zero, and left me with a score file I could copy back. I hashed both files on my laptop and felt a small, wrong relief when the digests matched exactly. Why would a planted bad case survive a live call unless the live call never happened at all?

python3 -m venv .venv
source .venv/bin/activate
python harness.py --cases cases.jsonl --out runs/latest.json --require-live
sha256sum runs/latest.json fixtures/expected_stub.json

On the second morning I stopped reading the exit code and started reading the environment the process actually inherited. I printed the working directory, the absolute output path, and whether the live base URL was empty. The empty URL sent the client into the fixture branch, and the fixture branch still returned success to the shell. Have you noticed how often a fallback is designed to stay quiet so the happy path can continue?

python - <<'PY'
import os
from pathlib import Path
print("cwd", Path.cwd().resolve())
print("base_url_set", bool(os.environ.get("MODEL_BASE_URL")))
print("out", Path("runs/latest.json").resolve())
PY

What actually broke

The remote shell was non-interactive, so the env file I had edited in my own session was never exported for the job. MODEL_BASE_URL was empty, the client treated that as a local dry run, and dry runs are allowed to exit zero. The score file I copied back was the stub I had committed, not a new live transcript from the remote process. I had compared two copies of the same decision, then congratulated myself for a consistency that meant nothing.

A second mistake sat in the wrapper I used for the download, and it was almost as embarrassing. The wrapper fell back to the committed stub whenever SCORE_OUT was unset, which is how that remote shell was configured. I hashed that fallback path, saw a match against itself, and called the night reproducible anyway. Which file did the hash command open, and which file did I think the remote job had written?

The check I would repeat

I do not need a vendor SDK for this check, and I am not describing an undocumented product response schema. The artifact below is my own harness contract, and a missing base URL is a failure rather than a polite stub. The score file must record source, run id, and the absolute output path so a later diff cannot lie about which file you read. I would rather fail closed in a minute than debug a green job for another long evening.

The harness contract

#!/usr/bin/env python3
"""Labeled local harness proposal you can run yourself, not a vendor client and not a measured bake-off."""
import argparse
import hashlib
import json
import os
import time
import uuid
from pathlib import Path

def normalize(text: str) -> str:
    return " ".join(text.strip().split())

def fixture_answer(case: dict) -> str:
    return case.get("stub", "stub-answer")

def live_answer(case: dict, base_url: str) -> str:
    # Replace this body with the HTTP call you actually intend to run on the live branch.
    # Keep the returned value as plain text so the scorer does not grow a second parser.
    raise RuntimeError(f"live call not configured for {base_url}")

def score_case(case: dict, answer: str) -> bool:
    expected = case.get("expect_contains", "")
    return expected.lower() in answer.lower()

def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--cases", required=True)
    parser.add_argument("--out", required=True)
    parser.add_argument("--require-live", action="store_true")
    args = parser.parse_args()
    base_url = os.environ.get("MODEL_BASE_URL", "").strip()
    source = "live" if base_url else "fixture"
    if args.require_live and source != "live":
        raise SystemExit("require-live set, but MODEL_BASE_URL is empty")
    run_id = os.environ.get("RUN_ID", str(uuid.uuid4()))
    out = Path(args.out).resolve()
    rows = []
    for line in Path(args.cases).read_text(encoding="utf-8").splitlines():
        if not line.strip():
            continue
        case = json.loads(line)
        if source == "live":
            answer = live_answer(case, base_url)
        else:
            answer = fixture_answer(case)
        rows.append({
            "id": case["id"],
            "source": source,
            "run_id": run_id,
            "ok": score_case(case, normalize(answer)),
            "answer": normalize(answer),
        })
    payload = {
        "source": source,
        "run_id": run_id,
        "out": str(out),
        "created": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
        "rows": rows,
    }
    out.parent.mkdir(parents=True, exist_ok=True)
    out.write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8")
    digest = hashlib.sha256(out.read_bytes()).hexdigest()
    print(f"source={source} run_id={run_id} sha256={digest} out={out}")
    if args.require_live and any(row["source"] != "live" for row in rows):
        raise SystemExit("fixture row leaked into a require-live run")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

A tiny fixture file keeps the failure mode visible when you review the command history later in the week.

{"id":"planted-fail","prompt":"Name the tool.","stub":"stub-answer","expect_contains":"this-will-not-match"}

Commands that should fail closed

Then the commands I actually want in the shell history are short, loud, and hard to misread after a weekend. The example.invalid host is deliberate, because this snippet must fail until you point it at a real base URL you control. I am not publishing an endpoint, a model name, a quota, or a hardware shape I was not given by the operator. If the free server option is what you use to reach that URL, the harness still only knows what your environment tells it.

export RUN_ID="$(date -u +%Y%m%dT%H%M%SZ)-$$"
export MODEL_BASE_URL="https://example.invalid/v1"
python harness.py --cases cases.jsonl --out "runs/${RUN_ID}.json" --require-live
python - "runs/${RUN_ID}.json" <<'PY'
import json, sys
doc = json.load(open(sys.argv[1], encoding="utf-8"))
assert doc["source"] == "live", doc["source"]
assert doc["run_id"]
print(doc["out"])
PY

A decision table, not a slogan

Question	Fixture on laptop	Free model, local process	Free model via free server
What am I proving?	Scorer and file format	Client can leave the stub	Remote env and client agree
Empty base URL	Allowed	Fail if require-live	Fail if require-live
May the score match a committed stub?	Yes	No	No
Good place for secrets?	No	No	No
Good place for a quality bake-off?	No	Not by itself	Not by itself

Read that table as a gate, not as a ranking of hosting options or a claim about which plan is faster. A free server is useful when the question is environmental, covering path, shell, env, and whether the live branch ran. A free model call is useful when the question is whether your client accepts a real response shape at all. Neither option, by itself, tells you that the answer text is good enough to ship to someone else.

What I would repeat next time

1. Print source, run id, absolute output path, and a sha256 before the process is allowed to exit zero.
2. Treat an empty live setting as a hard failure whenever the flag says the run must be live.
3. Store the score file outside the repo, or refuse to hash any path that sits inside a fixture directory.
4. Compare run ids, not just file bytes, so two successful stub runs cannot impersonate a remote experiment.
5. Keep the live call behind one function so a future client change cannot quietly restore the fallback.

I would also keep a one-line note in the run log that says which machine wrote the file. A laptop and a remote box can both say completed, and those words do not identify a host. Would a future me know which shell inherited the variable, or would I guess from memory again? I would rather see a hostname in the JSON than reconstruct the night from a vague completed line.

Limits, and who should skip this

This check does not measure answer quality, and it does not measure latency in a way you should quote. It will not notice a live model that returns fluent nonsense, as long as your expected substring happens to appear. It also will not survive a client that catches the live error and substitutes the stub after the source label was set. I left that hole visible on purpose, because a source label you set too early is another way to lie.

Do not use this approach for private customer data, production credentials, or anything you cannot stand to see in a log. Do not use it if you need a service-level agreement, a fixed quota, or a promise that a free option stays free. I was not given those numbers, so I will not invent them to make the note sound more certain. People who need a compliance review, a signed data-processing agreement, or deterministic grading of open-ended prose should pick a different setup.

The live function in the sample raises until you replace it, which is a limitation of the write-up rather than a hidden feature. If your real client returns tool calls, streams, or a different JSON shape, you must adapt the parser before you trust the source label. A green assert on a live source label only proves the branch, not the meaning of the text that branch returned. If you skip that adaptation, you will get a confident file that describes a protocol you never actually parsed.

What stayed with me

The expensive part was not the remote box, and it was not the model call I thought I was making. The expensive part was a quiet fallback that preserved a happy exit code while the score file described a different world. I would repeat the loud run record, the hard fail on an empty URL, and the refusal to hash a fixture path. If you already reach for MonkeyCode on short checks, prove the live branch ran before you argue about the answer.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.