Grade the Receipt, Not the Agent Demo

It is 11:52 PM and the submission is a nine-minute screen recording. You watch an agent read an issue, patch two files, run the test suite, and go green. You watch it twice. Then you close the tab, because you still do not know what the run cost, how many model calls it took, or whether any path other than the happy one was ever exercised.

The demo answers a narrow question: can this thing run once, on the author's laptop, with the author watching? That is not the question a take-home is supposed to answer. A take-home is a measurement instrument, and an instrument that only works while its owner holds it is not measuring anything.

Make the deliverable a receipt

When agent-shaped take-homes started showing up everywhere, the quality of the demos went up and the quality of the evidence went down. Candidates learned to make a loop that looks autonomous; reviewers learned to nod at a terminal recording. Nothing in that exchange proves the candidate can build something that stays inside a constraint.

So change the deliverable. Stop asking for a working demo and ask for a frozen ledger — a file the reviewer can recompute in under a minute and compare against what the candidate claimed. Everything else in the task hangs off that.

Here is the whole spec, short enough to paste into a message:

Build a tool that resolves fixtures/issue_142.json.
Hard budget: 40,000 tokens, 25 tool calls, 300 seconds wall clock.
Emit one JSON object per model call to usage.jsonl,
with prompt_tokens and completion_tokens.
If you hit a cap, stop and write give_up.json. A clean stop counts as a pass.
Deliver: agent.py, the usage.jsonl from your own run, and a ten-line note
naming one thing your agent cannot do.

Three numbers, one file format, one admission of weakness. That is the entire assignment.

The harness that makes the receipt checkable

You cannot grade a ledger you cannot regenerate. Write a small runner that executes the candidate's command under the same caps you published, then freezes the result. This is the version I hand to reviewers:

#!/usr/bin/env python3
"""budget_run.py — execute a submission under a fixed budget and freeze the receipt."""
import hashlib, json, pathlib, subprocess, sys, time

SPEC = {"token_cap": 40_000, "wall_seconds": 300, "tool_call_cap": 25}

def read_usage(path):
    if not pathlib.Path(path).exists():
        return 0, 0
    calls = [json.loads(l) for l in pathlib.Path(path).read_text().splitlines() if l.strip()]
    tokens = sum(c.get("prompt_tokens", 0) + c.get("completion_tokens", 0) for c in calls)
    return tokens, len(calls)

def digest():
    h = hashlib.sha256()
    for p in sorted(pathlib.Path(".").glob("*.py")):
        h.update(p.read_bytes())
    return h.hexdigest()[:16]

def main():
    started = time.monotonic()
    try:
        proc = subprocess.run(sys.argv[1:], capture_output=True, text=True,
                              timeout=SPEC["wall_seconds"])
        timed_out, code = False, proc.returncode
    except subprocess.TimeoutExpired:
        timed_out, code = True, None
    wall = round(time.monotonic() - started, 2)
    tokens, calls = read_usage("usage.jsonl")
    flags = [name for name, bad in {
        "tokens": tokens > SPEC["token_cap"],
        "wall_seconds": wall > SPEC["wall_seconds"],
        "tool_calls": calls > SPEC["tool_call_cap"],
        "missing_usage_file": calls == 0,
        "timeout": timed_out,
    }.items() if bad]
    ledger = {"cmd": sys.argv[1:], "wall_seconds": wall, "tokens": tokens,
              "tool_calls": calls, "exit_code": code, "over_budget": flags,
              "digest": digest()}
    pathlib.Path("ledger.json").write_text(json.dumps(ledger, indent=2))
    print(json.dumps(ledger, indent=2))
    return 1 if flags else 0

if __name__ == "__main__":
    sys.exit(main())

A reviewer runs one line, twice: once against the submitted artifact, once against a fixture the candidate has never seen. python budget_run.py python agent.py --task fixtures/issue_142.json. If the digest changes between runs, the candidate modified code between the claim and the check, and the conversation starts there instead of in the comments.

A sample solution that stops on purpose

The interesting part of a good submission is not the loop; it is the guard in front of the loop. This is roughly what a passing solution looks like, and the failure branch matters more than the success branch:

# agent.py — spend first, then act. Every model call leaves a receipt.
import json, pathlib, sys

CAP = 40_000
USAGE = pathlib.Path("usage.jsonl")

def spent():
    if not USAGE.exists():
        return 0
    return sum(json.loads(l)["prompt_tokens"] + json.loads(l)["completion_tokens"]
               for l in USAGE.read_text().splitlines() if l.strip())

def call_model(client, messages, tools):
    if spent() + estimate(messages) > CAP:          # estimate() is your tokenizer
        pathlib.Path("give_up.json").write_text(
            json.dumps({"reason": "token_cap", "spent": spent()}))
        sys.exit(0)                                  # clean stop, not a crash
    reply = client.chat(messages=messages, tools=tools)
    with USAGE.open("a") as f:
        f.write(json.dumps({"prompt_tokens": reply.usage.prompt_tokens,
                            "completion_tokens": reply.usage.completion_tokens}) + "\n")
    return reply

Notice that exhaustion is a first-class outcome. A candidate who wrote this has understood that budget is a design constraint; a candidate who retries fourteen times after a throttling response has not, and the ledger shows it without you having to argue about it.

The rubric, weighted for evidence

Keep it to five rows. Longer rubrics collapse into style grading.

Criterion	Weight	A pass looks like
Reproducibility	25	Your one-line invocation exits 0 on a clean checkout
Budget honesty	25	Your ledger.json and their claimed spend agree within 5%
Failure behavior	20	The over-budget run stops and writes give_up.json instead of thrashing
Held-out task	20	The agent clears a fixture it has never seen
Write-up	10	The ten-line note names a real limitation of its own approach

Everything above is checkable. Nothing depends on whether you liked the candidate's variable names.

The failure modes you will actually see

The first is the missing receipt: an agent that works, plus no usage.jsonl. That is not a formatting slip — it means the candidate never treated spend as part of the deliverable. The second is the retry storm, where a throttled call triggers a loop that quietly eats the entire cap. The third is fixture hardcoding, which is exactly why the held-out task carries twenty points.

The fourth is the interesting one: success by deletion. The agent "fixes" the failing test by removing the assertion. Grade the diff, not the exit code, and this one surfaces in seconds.

The fifth is machine drift, and it is the one worth designing out before you grade anyone. Give candidates a fixed environment instead of asking them to describe their laptop, because a submission that only runs on the author's box is not a submission. The open-source MonkeyCode project is one option here, and it is the one I point candidates at, since it offers a free tier and a hosted server so nobody's hardware decides their score. The project's outreach states a free allowance of ten million tokens plus a free server option; terms like that move, so read the current ones before you write a number into a rubric. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Limits, and who should skip this

This harness measures tokens, wall clock, and tool calls. It does not measure correctness — held-out tests still do that, and without them you are grading thrift. It also cannot prove the candidate didn't call some unlogged endpoint from a subprocess; it only makes hiding expensive and visible.

If your organization forbids third-party network access during interviews, do not use a hosted tier, free or otherwise. If you cannot freeze a fixture set, skip the held-out row and be honest that you are grading reproducibility only. And if your team has no fixed budget to publish, the whole exercise is theater — a take-home without a constraint is a portfolio review with extra steps.

If you want to dry-run the harness before you spend your own credits on it, start on MonkeyCode's free tier and check whether your ledger.json matches what the dashboard reports. A five-minute discrepancy hunt will teach you more about your rubric than another round of demo videos.

MonkeyCode provides free models that can run this workflow.