Grade the Tool Call, Not the Writeup

You unzip the take-home at 9:12. The README would make a staff engineer nod. There is a latency table with two decimal places. One paragraph swears the agent called an API, got a 200, and handled the error path.

You grep the archive for a request body, a tool name, a transcript that is not a paragraph. Nothing. The numbers are round. The timestamps are rounder. You have been handed a book report with a plot about a server.

That is the miss this assignment is built to catch. Not whether a model can sound senior. Whether a tool actually ran with an input you minted, and whether the packet still contains that run.

Flip a coin before you send the prompt

Before you mail anything, you mint a value the candidate cannot have memorized. Sixteen bytes of hex. You keep a copy. They do not get a second one. If their artifact cannot show that this exact string entered a tool and came back transformed, the essay is decoration.

# mint_nonce.py — run once per candidate, on your machine
import secrets
from datetime import datetime, timezone

nonce = secrets.token_hex(16)
issued_at = datetime.now(timezone.utc).isoformat()
print(f"{issued_at} {nonce}")
open("NONCE.txt", "w").write(nonce + "\n")

Think of it as a waiter’s ticket. Anyone can describe a kitchen. Only the ticket that left your hand and came back with grease on it proves an order was cooked. A trace id you did not issue can be invented. A nonce you issued cannot, unless they actually held it.

The prompt you actually hand them

Here is a take-home you can send as-is. Time-box it. Do not ask for architecture theater. The point is a tool call you can replay, not a strategy memo.

You have 45 minutes.

You are given NONCE=<paste the hex here>.
You must not hard-code a fake weather payload.

Implement a single tool named lookup_city that accepts
{ "city": string, "nonce": string }.

The tool must:
1. Reject a city that is empty or only whitespace.
2. Include the nonce on the outbound side as header X-Takehome-Nonce
   if you actually perform HTTP; if you stay in-process, still persist
   that header object next to the call.
3. Return JSON with:
   - echo_nonce: the nonce you received
   - nonce_sha256: SHA-256 hex of (nonce + ":" + city)
   - source: "tool" (literal)
   - city: the city argument

Then run an agent loop (model or a stub you label as a stub) that:
- looks up "Lisbon"
- refuses " "

Submit four files, nothing else:
- lookup_city implementation
- tool_call.json (the exact arguments the loop emitted)
- result.json (the exact tool result)
- timeout.md (one page: what you do if the tool never returns)

If any file is missing, the take-home is incomplete.
Do not submit screenshots.

You are not grading prose. You are grading whether a tool was specified, invoked, and answered with your coin still inside it. The timeout page is the only human writing you should need. Everything else a script can fail in seconds.

A rubric that refuses vibes

Score the packet, not the confidence. The table below is the whole interview loop for this assignment. Four of five gates are mechanical. The fifth is whether they even attempted the empty-city path.

Gate	Pass	Fail
Nonce echo	echo_nonce matches the issued hex, byte for byte	Missing, truncated, or a different string
Binding	nonce_sha256 equals SHA-256 of nonce + ":" + city for Lisbon	Hash of nonce alone, hash of city alone, or a made-up digest
Tool shape	One function lookup_city with both arguments	A free-form essay that "would call" an API
Refusal	Empty city does not produce a success object	Whitespace still returns a weather story
Transcript	Submitted JSON is valid and matches the implementation	Screenshots, paraphrases, or "the model said it worked"

A proposal grader you can run while the writeup tries to charm you:

# grade_takehome.py — proposal, unexecuted against your files until you run it
import hashlib
import json
import sys

issued = sys.argv[1]
city = "Lisbon"
result = json.load(open("result.json"))
call = json.load(open("tool_call.json"))

expected = hashlib.sha256(f"{issued}:{city}".encode()).hexdigest()

assert call.get("name") == "lookup_city", "wrong or missing tool name"
assert call.get("arguments", {}).get("nonce") == issued, "nonce never entered the call"
assert call.get("arguments", {}).get("city") == city, "city never entered the call"
assert result.get("echo_nonce") == issued, "nonce was not echoed"
assert result.get("nonce_sha256") == expected, "hash is not bound to city"
assert result.get("source") == "tool", "source was not the tool"
assert result.get("city") == city, "city mismatch"
print("automatic gates passed; now open timeout.md yourself")

If they never emitted {"city": " ", "nonce": "..."} in a second transcript, they skipped the path that shows up when an agent gets sloppy with arguments. That is not a style issue. That is the bug you will debug at 2 a.m.

A sample solution you can read in one sitting

This is a sample, not a production agent. It records the call the way the prompt asked, instead of summarizing it after the fact. If you wire a real model later, the only extra job is to dump the raw tool-call message before you execute it. Pretty logs that round-trip through a formatter are how nonces die.

# lookup_city.py
import hashlib

def lookup_city(city: str, nonce: str) -> dict:
    header = {"X-Takehome-Nonce": nonce}
    if not city or not city.strip():
        return {
            "error": "city_empty",
            "source": "tool",
            "echo_nonce": nonce,
            "headers": header,
        }
    digest = hashlib.sha256(f"{nonce}:{city}".encode()).hexdigest()
    return {
        "echo_nonce": nonce,
        "nonce_sha256": digest,
        "source": "tool",
        "city": city,
        "headers": header,
    }

# agent_loop.py — persist the call, then run the tool
import json
from lookup_city import lookup_city

def emit_tool_call(city: str, nonce: str) -> dict:
    # Label: in a real loop this JSON comes from the model.
    # You still write it to disk before lookup_city runs.
    return {
        "name": "lookup_city",
        "arguments": {"city": city, "nonce": nonce},
    }

if __name__ == "__main__":
    nonce = open("NONCE.txt").read().strip()
    call = emit_tool_call("Lisbon", nonce)
    result = lookup_city(**call["arguments"])
    open("tool_call.json", "w").write(json.dumps(call, indent=2))
    open("result.json", "w").write(json.dumps(result, indent=2))
    refused = lookup_city(" ", nonce)
    open("refusal.json", "w").write(json.dumps(refused, indent=2))
    print("wrote tool_call.json, result.json, refusal.json")

A tiny check for the refusal path, so you are not grading that one by eye:

python - <<'PY'
from lookup_city import lookup_city
out = lookup_city(" ", open("NONCE.txt").read().strip())
assert out["error"] == "city_empty"
assert "nonce_sha256" not in out
print("empty city refused")
PY

If you do involve HTTP, keep the nonce in a header you control and echo it in the body. A status code alone is a shrug. Anyone can type 200.

Where a free model and a free server actually help

You can run this loop on a laptop. Plenty of people should. The moment a model enters the loop, local keys and "I will paste the output later" start leaking into the take-home. Candidates stall. Agents invent tool results because the runtime never existed. The writeup gets prettier as the evidence gets thinner.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access and free server option are enough to execute the loop above without standing up paid inference or a box you babysit for a 45-minute assignment. Use that when the point of the exercise is "did a tool run," not "did we negotiate a cloud account." This article does not attach model names, quotas, hardware, or uptime promises; those change, and the rubric should not depend on them. Keep the nonce on your side. The remote box is a kitchen. The ticket still starts in your hand.

If you want a disposable runtime for the same 45-minute packet, that free server option is one place to try the loop without turning the take-home into an infra project.

How this usually breaks

The confident writeup with no tool_call.json is the classic miss. It reads well in a hurry. It cannot echo a coin it never held. You will be tempted to score the prose anyway. Do not.

The second miss is hashing the nonce alone. That proves they saw the string. It does not prove the city argument participated. Bind the two. Otherwise a template can replay last week’s digest and look exact.

The third miss is a screenshot of a chat window. Compression eats hex. You cannot grade pixels. Demand the JSON. If their tooling cannot export it, the tooling is part of the fail.

The fourth miss is executing lookup_city in the writeup and never letting the model, or the labeled stub, emit arguments. That is a unit test of a function you already have. The take-home is about the call. If nothing chose the tool, you learned nothing about the agent.

The fifth miss is leaking the nonce into a public gist and then being surprised when a later packet hashes it perfectly. Mint per candidate. Write the issued value next to the submission in notes you own. Expire it in those notes even if the server will not.

The sixth miss is a timeout page that is another invented latency table. You asked what they would do when the tool hangs. If the answer is "retry until it works" with no cap, no idempotency note, and no human fallback, they will hang you later.

Who should not use this

Do not use a nonce gate as a proxy for whether a human is smart. It measures whether an execution claim is grounded. That is a narrower question, and it is the only one this artifact answers. A brilliant candidate who never dumped the tool call still fails this packet. A weak agent that lucked into the hash still only proved the hash.

Skip it if your hiring process forbids sending unique tokens to candidates, or if you cannot store the issued nonce next to the zip. Skip it if the tool under test must run fully offline with no extra runtime; a free remote server will not make that constraint honest. Skip it if you need a vendor SLA, audit logs that survive a lawsuit, or a promise that a free tier will still be there on the day you interview. This workflow assumes a disposable 45-minute box and a script you own.

You flip the coin. You keep the other side. When the hash comes back bound to Lisbon, you have a take-home. When it does not, you have a story. Grade the first one.