You open a take-home at 11 p.m. and the README is a novella. Three screenshots of a chat panel sit under a heading called Demo. The last bubble reads Order created. There is no URL, no request body, no status code. You cannot tell if a model called anything, or if someone typed a happy path into Markdown and went to bed.
That is not a candidate problem. It is an assignment problem. If the role includes models that call APIs, a writeup is a story about a wire you never saw. You need a file you can replay.
The job moved. Most take-home zips did not. They still ask for a feature and a paragraph, the same shape that worked when the work was a CRUD app on localhost. It fails when the interesting part is a tool call. The model must pick a function, fill arguments, hit your HTTP boundary, and react to the status code that comes back. Chat logs lie about that loop. They summarize. They omit the 429. They never show that the Idempotency-Key on the retry did not match the first attempt.
You do not need a bigger rubric. You need the candidate to submit the trace. Think of it like a flight recorder. You do not grade the pilot's essay about the landing. You pull the box and look at airspeed. A JSONL of HTTP request and response is that box. If they cannot produce one, they did not close the loop. They narrated it.
Keep the prompt short enough to read on a phone. Paste something in this shape and label it as the assignment, not as a war story from a team you have not named.
Build a tiny shop stub and a harness that drives it through a model.
The stub exposes:
POST /orders
header Idempotency-Key required
JSON body: {"sku": string, "qty": int}
201 on first unique key; body includes order_id and trace_id
409 if the same key arrives with a different body
200 if the same key arrives with the same body (replay)
GET /orders/{id}
GET /_trace/{trace_id} — the exact request/response the stub stored
The harness:
- talks to a model over HTTP (base URL and model name from env)
- exposes one tool: create_order(sku, qty)
- must actually POST to the stub; do not invent the 201
- writes replay.jsonl, one object per HTTP hop, including trace_id
Submit:
1. replay.jsonl
2. the stub and harness
3. a base URL where the stub is still reachable for 48 hours
We will replay two rows from your JSONL against your URL.
If they do not match, the take-home is incomplete.
You are not asking them to train a model. You are asking them to prove a round trip. That is closer to production than another todo API, and it is still small enough for a weekend. The trick is to ship a stub that remembers what it saw, then refuse to score anything the stub cannot echo back.
The following stub is a reference solution, not a transcript from a billed service. It keeps orders and traces in memory, stamps X-Trace-Id, and lets you fetch that trace later. Read it as a starting point the candidate may replace, as long as /_trace/{id} still tells the truth.
# stub_shop.py — reference stub for the take-home
from http.server import BaseHTTPRequestHandler, HTTPServer
from urllib.parse import urlparse
import json, uuid, threading
LOCK = threading.Lock()
ORDERS = {} # order_id -> record
KEYS = {} # idempotency_key -> order_id
TRACES = {} # trace_id -> blob
class Shop(BaseHTTPRequestHandler):
def log_message(self, fmt, *args):
return
def _read(self):
n = int(self.headers.get("Content-Length", 0))
return json.loads(self.rfile.read(n) or b"{}")
def _send(self, code, payload, trace_id):
raw = json.dumps(payload).encode()
self.send_response(code)
self.send_header("Content-Type", "application/json")
self.send_header("X-Trace-Id", trace_id)
self.send_header("Content-Length", str(len(raw)))
self.end_headers()
self.wfile.write(raw)
def do_POST(self):
if urlparse(self.path).path != "/orders":
return self._send(404, {"error": "nope"}, str(uuid.uuid4()))
key = self.headers.get("Idempotency-Key")
body = self._read()
trace_id = str(uuid.uuid4())
bad = (not key or "sku" not in body or not isinstance(body.get("qty"), int))
if bad:
blob = {"path": "/orders", "key": key, "body": body, "status": 400}
with LOCK:
TRACES[trace_id] = blob
return self._send(400, {"error": "bad request", "trace_id": trace_id}, trace_id)
with LOCK:
if key in KEYS:
existing = ORDERS[KEYS[key]]
status = 200 if existing["body"] == body else 409
blob = {"path": "/orders", "key": key, "body": body,
"status": status, "order_id": existing["order_id"]}
TRACES[trace_id] = blob
else:
order_id = "ord_" + uuid.uuid4().hex[:8]
ORDERS[order_id] = {"order_id": order_id, "body": body, "key": key}
KEYS[key] = order_id
status = 201
blob = {"path": "/orders", "key": key, "body": body,
"status": status, "order_id": order_id}
TRACES[trace_id] = blob
order_id = blob.get("order_id")
payload = {"trace_id": trace_id, "order_id": order_id, "ok": status in (200, 201)}
if status == 409:
payload = {"trace_id": trace_id, "error": "idempotency conflict"}
self._send(status, payload, trace_id)
def do_GET(self):
parts = urlparse(self.path).path.strip("/").split("/")
trace_id = str(uuid.uuid4())
if len(parts) == 2 and parts[0] == "_trace":
rec = TRACES.get(parts[1])
if not rec:
return self._send(404, {"error": "unknown trace"}, trace_id)
return self._send(200, rec, trace_id)
if len(parts) == 2 and parts[0] == "orders":
rec = ORDERS.get(parts[1])
if not rec:
return self._send(404, {"error": "unknown order"}, trace_id)
return self._send(200, rec, trace_id)
self._send(404, {"error": "nope"}, trace_id)
if __name__ == "__main__":
HTTPServer(("0.0.0.0", 8080), Shop).serve_forever()
Start it with python stub_shop.py. Then prove the stub without a model at all. That sanity check belongs in the zip. Candidates who skip it usually invent the JSONL later, and you will feel it when /_trace 404s.
# first create -> 201 and an X-Trace-Id
curl -sS -D - http://127.0.0.1:8080/orders \
-H 'Content-Type: application/json' \
-H 'Idempotency-Key: k1' \
-d '{"sku":"sku_light","qty":2}'
# same key, same body -> 200
curl -sS -D - http://127.0.0.1:8080/orders \
-H 'Content-Type: application/json' \
-H 'Idempotency-Key: k1' \
-d '{"sku":"sku_light","qty":2}'
# same key, different body -> 409
curl -sS -D - http://127.0.0.1:8080/orders \
-H 'Content-Type: application/json' \
-H 'Idempotency-Key: k1' \
-d '{"sku":"sku_light","qty":9}'
Copy X-Trace-Id from the first response and hit /_trace/<that-id>. If that round trip does not work on localhost, it will not work after they deploy. You are teaching them the scoring move before they ever call a model. That is kindness, not a leak. A take-home that hides the grading wire is just a riddle.
The harness is the other half of the sample. It should refuse to write a success row unless the stub returned one. Environment variables keep vendor names out of the zip. Point it at any OpenAI-style chat endpoint. The script below is a labeled reference driver. It is not a claim that it was run against a paid account.
# harness.py — reference driver
import json, os, uuid, urllib.request, urllib.error
STUB = os.environ["STUB_URL"].rstrip("/")
MODEL_URL = os.environ["MODEL_BASE_URL"].rstrip("/")
MODEL = os.environ["MODEL_NAME"]
REPLAY = open("replay.jsonl", "a")
TOOLS = [{
"type": "function",
"function": {
"name": "create_order",
"description": "Create an order in the shop stub",
"parameters": {
"type": "object",
"properties": {
"sku": {"type": "string"},
"qty": {"type": "integer"}
},
"required": ["sku", "qty"]
}
}
}]
def hop(method, url, body=None, headers=None):
data = None if body is None else json.dumps(body).encode()
hdrs = {"Content-Type": "application/json"}
if headers:
hdrs.update(headers)
req = urllib.request.Request(url, data=data, method=method, headers=hdrs)
try:
with urllib.request.urlopen(req, timeout=20) as resp:
raw = resp.read()
rec = {
"url": url, "method": method, "status": resp.status,
"req": body, "resp": json.loads(raw.decode() or "{}"),
"trace_id": resp.headers.get("X-Trace-Id"),
}
except urllib.error.HTTPError as e:
raw = e.read()
rec = {
"url": url, "method": method, "status": e.code,
"req": body, "resp": json.loads(raw.decode() or "{}"),
"trace_id": e.headers.get("X-Trace-Id"),
}
REPLAY.write(json.dumps(rec) + "\n")
REPLAY.flush()
return rec
def create_order(sku, qty):
key = "idem_" + uuid.uuid4().hex[:12]
return hop(
"POST", f"{STUB}/orders",
{"sku": sku, "qty": qty},
{"Idempotency-Key": key},
)
def call_model(messages):
payload = {"model": MODEL, "messages": messages, "tools": TOOLS}
rec = hop("POST", f"{MODEL_URL}/chat/completions", payload)
return rec["resp"]
if __name__ == "__main__":
messages = [
{"role": "system", "content": "Create shop orders via tools. Never pretend the order exists."},
{"role": "user", "content": "Order 2 units of sku_light."},
]
first = call_model(messages)
msg = first["choices"][0]["message"]
messages.append(msg)
for tool_call in msg.get("tool_calls") or []:
args = json.loads(tool_call["function"]["arguments"])
result = create_order(**args)
messages.append({
"role": "tool",
"tool_call_id": tool_call["id"],
"content": json.dumps(result["resp"]),
})
final = call_model(messages)
print(json.dumps({"final": final, "replay": "replay.jsonl"}, indent=2))
The important line is not the model call. It is REPLAY.write. If the model hallucinates an order_id, that id will not show up under /_trace on the stub. You will notice. That is the whole point of making the server the source of truth instead of the chat transcript.
Sit down with two terminals, not with their README. Pull two trace_id values from replay.jsonl. Curl their still-live stub. Compare JSON. You are grading three questions only. Did the model emit a tool call, or did they paste a 201 into the log by hand? Did the stub enforce idempotency, or is every POST a new order? Do the bytes on the wire match the file they zipped?
A writeup can still exist. It just cannot replace those three checks. If you want a paper trail for the hiring committee, record the curls you ran. That recording is yours. Their screenshots are not. A candidate who argues that the chat screenshot is easier to read is telling you they optimized for looking finished. You are hiring for a closed loop.
The collapses rhyme once you have scored a few of these. The JSONL contains only the model chat and no stub hops, which means they logged the narrator and not the tool. The stub still binds 127.0.0.1 and they forgot a public URL, so your Monday replay dies in DNS. They log status codes but drop Idempotency-Key, so you cannot tell a 200 replay from a lucky second insert. They let the model invent order_id when the stub is down, and the harness never fails closed. They rotate a fresh URL after you start scoring. They submit a 400-line architecture novel and a three-line trace.
Each of those is a signal, not a personality test. Missing keys is a product bug. A dead URL is an ops miss, and you should say so in the prompt if process restarts wipe in-memory maps. A hallucinated success row is the one you cannot hire for a tool-calling role. That is how phantom orders reach finance. A beautiful stub that never returns 409 is also a fail, because they deleted the interesting branch to make the demo pretty.
You should not require a paid API key to finish this assignment. The candidate is already giving you a weekend. If the harness reads MODEL_BASE_URL and the stub binds 0.0.0.0, they can point both at something that costs them nothing and still leave you a URL to curl.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
MonkeyCode offers free model access and a free server option. That pair is enough to host the stub at a public URL and to drive the harness without pasting a vendor key into the zip. Use it as plumbing. It does not grade the trace for you, it does not promise the model will call the tool on the first try, and it does not replace the two curl checks you run on Monday. If you already have a lab box and an internal model gateway, skip the product. The assignment does not depend on it.
Do not ship this zip for a role that never touches HTTP. You will filter for the wrong muscle. Do not use it if your legal team forbids any model in take-homes, because the harness is explicit about calling one. Do not use it as live onsite theater. The value is the URL that still answers after the candidate has slept, not the performance of watching someone type.
Treat the stub as a toy. In-memory maps die on restart, so a free process that recycles will empty /_trace. Say that in the prompt. Ask for the JSONL anyway, and treat a 404 on replay as a deployment note unless the file itself is empty or internally inconsistent. Do not claim the method measures reasoning. It measures whether a tool call happened and whether the candidate can keep a server honest long enough for you to look.
You will still read the code. A perfect trace with the 409 path commented out is a no. A messy trace with an honest conflict and a retry is a yes. The file is evidence. It is not the hire. When you close the laptop this time, you want two matching JSON objects, not a screenshot of a chat bubble that says Order created. If they cannot replay it, you did not grade it. You read a story.
Top comments (0)