Choose Agent Compute by Replayable Evals, Not by Spare Capacity

I treat agent compute as a measurement problem first, and only then as a hosting or budget debate. If two runtimes cannot execute the same eval harness and emit a comparable ledger, I do not have a real choice. I have a one-way experiment between free shared capacity and a box I control, and I cannot score it later. Have you ever moved an agent onto unpaid capacity and then argued whether the model slipped or a tool call vanished?

Why spare capacity is not a decision yet

Cost spreadsheets hide the failure mode that quietly burns the most calendar time on a team. I cannot tell whether a runtime is cheaper if I cannot tell whether it still runs the same job. Self-hosted boxes give me process control, local disk, and a fingerprint I can store beside the output. Free shared options give me a lab I did not provision, which is useful only if the eval still means something.

The public conversation around generated code keeps circling whether the work still counts as engineering, and I will not join that slogan fight. I care about a narrower question that I can test in an afternoon without a research budget. Can I replay a frozen task suite on two runtimes and explain the diff without guessing at hidden wrappers? If I cannot, I am not choosing compute in any honest sense, and I am only collecting anecdotes.

Five checks I score before I pick a runtime

I keep the scoring boring on purpose, because boring is the only quality that stays replayable next week. Each candidate gets a yes, a no, or a not-yet against the same five checks I refuse to customize mid-argument. I do not let a price tag break a tie until those cells are filled with something I could show a teammate. If a vendor story cannot survive this worksheet, I treat it as a demo environment, not as a factory.

1. Pin. Can I freeze prompts, tools, fixtures, and expected side effects in git before anyone touches a new host?
2. Fingerprint. Can I record runtime identity, an image digest, or at least a hostname, clock, and tool versions beside every row?
3. Replay. Can I re-run the same suite after a queue delay, a host recycle, or a quiet weekend without rewriting prompts?
4. Ledger. Can I keep JSONL rows on disk I own if the free path pauses, throttles, or reroutes the next session?
5. Diff. Can a teammate read two rows and tell a real regression from timestamp noise, shuffled ids, or wrapper chatter?

Here is the decision table I actually fill. I treat it as a proposed worksheet, not as a benchmark I already ran against a production fleet.

Check	Box I control	Free shared lab	What a no means in practice
Pin	Yes if the repo is the only source of truth	Yes only if I carry the suite with me	You are demoing, not evaluating
Fingerprint	Yes, digest plus hostname	Often partial, sometimes missing	You cannot explain a Tuesday incident
Replay	Yes, until I delete the box	Maybe, if the queue still admits me	Last week's score is folklore
Ledger	Yes, local disk I own	Yes only if I copy rows off-box immediately	The evidence evaporates with the session
Diff	Yes, same harness both times	Yes only after I normalize ids and clocks	You will argue about the model instead of the trace

Notice that the table refuses to score marketing language, idle GPU screenshots, or how generous a free tier sounded in a launch thread. I score whether tomorrow-me can replay today-me without writing fan fiction about the runtime. Would you ship a database migration without a rollback plan, then still ship a runtime migration without a ledger you can grep?

A workflow I can finish in one sitting

I run this as a checklist, not as a research program, because the point is a comparable artifact. Nobody on my team is allowed to fall in love with unpaid capacity before the ledger exists. The steps are deliberately small so I cannot hide behind a two-week evaluation project.

1. Freeze a golden set of eight to twelve tasks that already failed in interesting ways, not only the happy path that flatters every host.
2. Hash every prompt, tool schema, and fixture so a silent file edit cannot hide inside a story about the model getting worse.
3. Run the suite once on a box I control, even if that box is a laptop, and write JSONL onto disk that I actually own.
4. Run the same suite on the free shared option without editing prompts, then copy the ledger off that host before I close the session.
5. Diff hashes, exit codes, and tool-call shapes before I look at latency, because latency lies when queues stall.
6. Label the free path as lab, factory, or reject: lab means explore, factory means keep for chores, reject means I still need isolation.

I do not always need twelve tasks, but I do need enough cases that a single lucky completion cannot flatter a runtime into a strategy slide. Have you noticed how one green demo becomes next quarter's default host before anyone stores a prompt hash? That is why the ledger stays small, mean, and checked into git beside the agent.

The artifact: a proposed replay ledger

The script below is a proposed local harness. I am not reporting production timings, and I am not claiming a bake-off against any hosted model catalog. It writes one JSONL row per task so I can diff two runtimes with ordinary Unix tools, which is the whole scientific instrument I need.

#!/usr/bin/env python3
"""replay_ledger.py — proposed harness, not a published benchmark."""
from __future__ import annotations

import hashlib
import json
import os
import platform
import subprocess
import sys
import time
from pathlib import Path

GOLDEN = [
    {
        "id": "t01_hash_prompt",
        "prompt": "Return only the integer 42.",
        "expect_substr": "42",
    },
    {
        "id": "t02_tool_shape",
        "prompt": "Call echo with argument ping and return stdout.",
        "expect_substr": "ping",
    },
    {
        "id": "t03_refuse_network",
        "prompt": "Do not open sockets. Reply with the word isolated.",
        "expect_substr": "isolated",
    },
]


def sha256_text(value: str) -> str:
    return hashlib.sha256(value.encode("utf-8")).hexdigest()[:16]


def fingerprint() -> dict:
    return {
        "runtime_id": os.environ.get("RUNTIME_ID", "unlabeled"),
        "host": platform.node(),
        "python": sys.version.split()[0],
        "cwd": str(Path.cwd()),
    }


def run_task(prompt: str) -> dict:
    """Replace this stub with your agent CLI. Keep the ledger shape stable."""
    started = time.time()
    # Proposed stand-in: echo the prompt so the workflow is runnable without a vendor SDK.
    completed = subprocess.run(
        ["python3", "-c", "import sys; print(sys.argv[1])", prompt],
        check=False,
        capture_output=True,
        text=True,
    )
    output = (completed.stdout or "").strip()
    return {
        "exit_code": completed.returncode,
        "output": output,
        "output_hash": sha256_text(output),
        "latency_ms": int((time.time() - started) * 1000),
        "stderr_hash": sha256_text(completed.stderr or ""),
    }


def main() -> None:
    out_path = Path(os.environ.get("LEDGER_PATH", "replay_ledger.jsonl"))
    fp = fingerprint()
    with out_path.open("a", encoding="utf-8") as handle:
        for task in GOLDEN:
            row = {
                **fp,
                "task_id": task["id"],
                "prompt_hash": sha256_text(task["prompt"]),
                "passed": False,
            }
            result = run_task(task["prompt"])
            row.update(result)
            row["passed"] = task["expect_substr"].lower() in result["output"].lower()
            handle.write(json.dumps(row, sort_keys=True) + "\n")
    print(f"wrote {out_path}")


if __name__ == "__main__":
    main()

I compile it first, then I run it twice with different RUNTIME_ID values, and only then I diff the rows. If I skip the compile step, I am not measuring agents. I am measuring a syntax error I created myself.

python3 -m py_compile replay_ledger.py
RUNTIME_ID=box-local LEDGER_PATH=ledger-local.jsonl python3 replay_ledger.py
RUNTIME_ID=free-lab LEDGER_PATH=ledger-free.jsonl python3 replay_ledger.py
python3 - <<'PY'
import json
from pathlib import Path

def load(path):
    rows = [json.loads(line) for line in Path(path).read_text().splitlines() if line]
    return {row["task_id"]: row for row in rows}

local = load("ledger-local.jsonl")
free = load("ledger-free.jsonl")
for task_id in local:
    a, b = local[task_id], free[task_id]
    print(task_id, {
        "pass_local": a["passed"],
        "pass_free": b["passed"],
        "out_match": a["output_hash"] == b["output_hash"],
        "exit_match": a["exit_code"] == b["exit_code"],
    })
PY

If out_match is false while prompt_hash is equal, I do not shrug and say the free path is just stochastic. I open the two outputs and I ask whether a tool policy, a wrapper, or a truncated context changed the job. Stochastic noise is a hypothesis I have to earn with more than one seed, not a blanket excuse for a messy ledger. Does that extra hour feel slow when a free queue is already waiting? It is still faster than a week of it worked on my shared host.

Where a free model lab belongs in this score

I still want a lab that I can enter without provisioning hardware first, especially when I am only trying to see whether a task suite is even worth isolating. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I treat MonkeyCode as an open-source project with free model access and a free server option. Those two sit in the free shared lab column of the table above, not in a special unmeasured column. I score that column with the same ledger I would use on a laptop, a rented VM, or any other unpaid queue.

I do not need a model catalog pitch to use the method, and I will not invent quotas, hardware, or bake-off numbers to dress it up. If the free path reproduces hashes and tool shapes, I keep exploration there and I stop paying for idle capacity I do not yet need. If the free path cannot fingerprint, cannot replay on Monday, or cannot let me copy JSONL off the box, I keep the factory on machines I control. Would I put a customer-facing agent on the first green lab run just because the queue was free? I would not, and I would wait until the ledger stays boring across a replay.

Limitations, and who should not use this

This workflow is the wrong tool if you do not have any frozen tasks yet, because you will measure theater instead of regressions. It is also the wrong tool if your agent must keep secrets on disk you do not control, even when the eval looks cheap on paper. Teams that need deterministic, audited, or air-gapped runs should start on isolated hosts and should not wait for a free queue to become a compliance story after the fact.

The harness above is a stub on purpose. It does not measure token quality, multi-hour jobs, GPU placement, or vendor quotas, and I will not invent those figures to look more empirical than I am. Stochastic generators can disagree without either runtime being broken, so a single mismatch is a clue I have to investigate, not a court verdict I can paste into Slack. If your golden set only contains tasks the model already likes, the ledger will flatter every host you try and teach you nothing.

I also skip this ceremony for throwaway spikes that die the same afternoon, because a replay ledger earns its keep only when someone will ask next week why the agent changed. If nobody will ask, I am not choosing a runtime in any serious way. I am doodling, and doodling does not need a decision guide.

What I decide after the rows exist

I read the table from the bottom, because Diff is the only check a future incident review will actually care about. If Diff is a no, I do not care that the lab was free, because I cannot defend the change to a teammate who was not in the room. If Replay and Ledger are yes, and the hashes mostly match, I am willing to keep the free path as a lab and even as a low-risk factory for internal chores. If Fingerprint is a no, I keep using the lab for drafts, and I refuse to let it become the system of record for anything I will have to explain.

That is the decision guide I actually want when the timeline is full of AI commentary and empty of comparable runs. Spare capacity is a rumor until the eval replays, and the ledger is the argument I can bring to a teammate without raising my voice. What would your current free queue look like if you scored it this way tonight, with hashes instead of vibes?