DEV Community: Morgan Zhou

Make the Trace the Deliverable

Morgan Zhou — Thu, 08 Oct 2026 09:19:11 +0000

Last month a candidate sent back a take-home that looked like a shipped product. Clean modules, a test suite, a README with an architecture diagram. Then I asked them to change one constraint live — run the same questions over a different folder — and the whole thing fell apart in about four minutes. Nothing in the repo showed how the model had decided anything.

That story is not rare right now. Look at what has been circulating this week: portfolios nobody has to build by hand anymore, agent writeups where the interesting detail is what a model noticed and then chose not to report. Both point at the same hiring problem. The artifact is cheap. The process is not.

So grade the process. Below is the take-home I've been handing out for tool-using agent roles, the grader that goes with it, and the ways people break it.

The prompt

Give a fixed corpus and exactly one endpoint. No framework, no scaffold — a budget and a required output format. The whole thing fits on one screen.

Build a small tool-using agent that answers questions over ./corpus
(plain text, roughly 40 files, no external index).

Constraints:
- One model endpoint, passed as MODEL_BASE_URL and MODEL_API_KEY
- At most 8 tool calls per question
- Emit trace.jsonl: one JSON object per step, in order
- No network calls outside the endpoint
- Also answer questions.jsonl (15 questions, 3 of which are not answerable)

Deliver: repo + trace.jsonl + a 300-word note on what you would fix
with another day.
Timebox: 3 hours. If you use a codegen assistant, say so in the note.

Every constraint earns its place. A single endpoint makes runs comparable across candidates. The eight-call cap forces tool selection to become visible; a brute-force grep sweep eats the budget and you can see it. The unanswerable questions check whether the agent ever says "not in the corpus" instead of inventing a clause. And the timebox plus the assistance disclosure keeps the exercise honest without pretending codegen does not exist.

One practical wrinkle: if candidates have to pay for tokens out of pocket, you are quietly ranking wallets. Hand everyone the same starting line. MonkeyCode is an open-source agent project that offers free model access and a free server option, and the operator states the free tier includes a generous token allowance — 10 million tokens as currently described. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Before you bolt any of this into a real pipeline, read the project's current terms yourself, because allowances and limits move and a number I quote today is not a contract.

The trace is the real submission

The repo shows what they built. The trace shows how they think. Ask for a flat JSONL file with three step types, and suddenly every submission is machine-readable and reviewable in one pass.

{"step":1,"type":"tool_call","tool":"grep","args":{"pattern":"refund window","path":"corpus"},"latency_ms":41}
{"step":2,"type":"tool_result","tool":"grep","bytes":812,"rows":3}
{"step":3,"type":"tool_call","tool":"read","args":{"path":"corpus/returns.txt","offset":120,"limit":40}}
{"step":4,"type":"final","answer":"30 days from delivery","citations":["corpus/returns.txt"]}

Format rules matter more than they look. Fixed step types mean a grader can run without an LLM in the loop, which means it costs nothing and gives the same answer twice. The citations field is the part candidates underestimate; a citation that does not resolve to a file, or resolves to a file that does not contain the answer, is worse than no citation at all.

The grader

This is the shape of the script I hand reviewers. Treat it as a template, not a validated benchmark — I have not run it across a cohort large enough to quote failure rates, and I would distrust anyone who did in a blog post.

import json, pathlib

KEY = {"q07": "30 days", "q12": "not in the corpus"}

def grade(trace_path, question_id, corpus):
    steps = [json.loads(l) for l in pathlib.Path(trace_path).read_text().splitlines() if l.strip()]
    calls = [s for s in steps if s["type"] == "tool_call"]
    final = next((s for s in steps if s["type"] == "final"), None)

    signatures = {(c["tool"], json.dumps(c["args"], sort_keys=True)) for c in calls}
    return {
        "within_budget": len(calls) <= 8,
        "stopped": final is not None,
        "citations_resolve": bool(final) and all((corpus / c).exists() for c in final.get("citations", [])),
        "answer_matches_key": bool(final) and KEY.get(question_id, "") in (final.get("answer") or ""),
        "duplicate_calls": len(calls) - len(signatures),
    }

Read the output as a profile, not a score. duplicate_calls above zero on an easy question usually means the agent ignored a tool result and retried the same query — a control-flow bug, not a knowledge gap. within_budget false plus a correct answer tells you the person brute-forced their way out of designing a search step.

The rubric

Signal	Weak	Strong
Budget use	Always 8 calls	Varies with question difficulty
Citation	Cites README or nothing	Cites the clause that contains the answer
Recovery	Aborts after a failed tool call	Narrows the query and continues
Unanswerable question	Invents a plausible answer	Says the corpus does not cover it
Trace honesty	Steps logged at the end, out of order	Written as it runs

The last row is the one that catches the most people. If timestamps and ordering look retrofitted, the trace is fiction. I usually re-run question one live in the follow-up call; an agent whose trace was generated after the fact almost never reproduces its own step shapes.

Common failure modes

The most frequent submission is the grepper: the agent searches the whole corpus on every question because it never learned to narrow. Its trace is monotonous and its budget is always exactly eight.

The second is the confident summarizer. It reads one file, generalizes to all forty, and produces a fluent answer with a citation that resolves to something tangentially related. This one survives a casual review and dies the moment you open the cited line.

The third is the hardcoder. Twelve answers, memorized, with no tool calls worth mentioning. This is exactly why you hold back three questions and run them yourself during the debrief instead of accepting the batch they submitted. A memorized solution cannot answer a question it has never seen.

The fourth is structural: logging bolted on at the end, offsets missing, final emitted twice. It reads less like dishonesty and more like someone who never expected the trace to be the deliverable. Tell them in the prompt, twice, that it is.

A sample solution skeleton

Not a reference answer — a shape. The interesting decisions live in the system prompt, not the loop.

def answer(question, tools, model, budget=8):
    ctx = [{"role": "system", "content": SYSTEM}]  # SYSTEM: cite files, stop at 8, say "not in corpus"
    for _ in range(budget):
        msg = model.chat(messages=ctx + [{"role": "user", "content": question}])
        if msg.tool_call:
            result = tools[msg.tool_call.name](**msg.tool_call.args)
            ctx += [msg.raw, {"role": "tool", "content": result}]
            continue
        return msg.text
    return "budget exhausted"

Two lines there carry most of the signal: the system prompt's stopping rule and the citation requirement. A candidate who spends their note explaining why they chose those two lines is telling you something a passing test run never will.

Limitations, and who should skip this

This only works when answers are checkable against a key. For design-heavy roles, a written proposal graded by a human is still the better instrument, and no amount of JSONL will fix that. Setup also costs more than a LeetCode screen — you write the corpus, the key, and the grader before the first candidate ever sees it.

Free tiers have terms, and terms change. Do not build a hiring process on a quota you read in someone's article, including this one. And if you cannot spare a reviewer hour for the debrief, skip the take-home entirely; an unreviewed submission is just unpaid work.

If the reproducibility angle is what appeals to you, the free model access and free server option in MonkeyCode is the least interesting part of it. The useful part is that everyone starts from the same endpoint and every run leaves a trace you can re-read six months later. Check the current terms, then decide.

Rerun the Take-Home on a Clean Machine

Morgan Zhou — Thu, 08 Oct 2026 06:37:32 +0000

You join the debrief with the recording still warm. The candidate walked a green script, named three files, and said the helper would flag a risky diff. You clone the repo after the call.

The same command stops on a missing key. The demo was a stage. The take-home was a suitcase you cannot open.

That gap is the assignment. You are not hiring a narrator. You are hiring someone who can leave a run a stranger can repeat. When the allowed path includes free model access and a free server, the stranger is you, tomorrow, on a clean machine, with none of their environment variables.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Those two availability claims, free model access and a free server, are the constraints this workflow uses.

This article does not pin model names, token quotas, hardware, duration, or permanence. Those details move. A rubric that freezes a number you have not verified will punish the wrong person. The packet should record the route the candidate actually called.

Think of the take-home as a lab sample, not a poster. A poster looks finished under good light. A sample has a label, a seal, and a way to run the same test again. You grade the seal, then the second run.

The prompt you send

You send a short assignment, not a puzzle novel. The text below is a proposal for your loop, not a transcript of a round that already happened.

Build a small review helper that reads fixtures/risky.diff and writes out/review.json. The JSON must include risk, set to low or high, and reason, one sentence. The candidate may call a model. They may run the helper on a free server if they have one.

They may not call any other network host. They commit a packet.json that tells you how to replay the run with no secrets from their laptop. Time box: four hours. You grade the replay, not a video.

That last line does the cultural work. Candidates who optimized for a screen recording will feel the floor move. You want the floor to be a command.

Own the fixture. A short diff that adds a shell call built from request input is enough. You are not asking them to invent an exploit, and you should not grade them as if they had. You are asking whether the helper flags that pattern, and whether you can reproduce the flag after they log off.

Commit this fixture yourself. Hash it before you send the repo. Your copy stays the oracle.

diff --git a/app.py b/app.py
--- a/app.py
+++ b/app.py
@@ -1,3 +1,6 @@
+import subprocess
 def handle(request):
-    return "ok"
+    cmd = request.args.get("cmd")
+    subprocess.run(cmd, shell=True)
+    return "ok"

The dangerous line is ordinary on purpose. Live demos love exotic bugs because exotic bugs photograph well. A replay loves a stable oracle. If the text contains a request-fed shell=True, the expected risk is high. Anything else on this fixture is a miss.

The seal on the sample

packet.json is the label. Keep it small enough to read without booking a meeting.

{
  "command": "python review.py --fixture fixtures/risky.diff --out out/review.json",
  "fixture_sha256": "replace-with-your-hash",
  "expected_risk": "high",
  "model_route": "declared-by-candidate-or-offline",
  "server": "local",
  "network_allow": [],
  "secrets_required": false
}

secrets_required must be false. If the replay needs a key that lives only in their shell profile, the take-home is unfinished. Free model access helps only when they can point the helper at a route you are willing to supply for the replay, or when an offline rule still flags the diff.

Do not write a token budget into the rubric. Write this instead: the declared route is reachable under access you control, or the offline path passes. A number you copied from a launch post is not an oracle.

The server field is the same kind of claim. A free server is a place to run the helper, not a sticker on the README. If they used one, the packet names it and the command works there.

If they stayed local, server is local. You do not punish a local run. You punish a packet that advertises the free server while the command only succeeds on a laptop path you cannot see.

A grader you run in a sandbox

The script below is a sketch. Run it on a machine that holds no customer keys. Candidate code is still code you did not write. Path.unlink(missing_ok=True) needs Python 3.8 or newer.

#!/usr/bin/env python3
"""Proposal grader. Not a log of an executed interview."""
import hashlib, json, subprocess, sys
from pathlib import Path

def sha256(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()

def main() -> int:
    root = Path(sys.argv[1]).resolve()
    packet = json.loads((root / "packet.json").read_text())
    if packet.get("secrets_required"):
        print("FAIL secrets_required")
        return 1
    fixture = root / "fixtures" / "risky.diff"
    if not fixture.is_file() or sha256(fixture) != packet["fixture_sha256"]:
        print("FAIL fixture hash")
        return 1
    out = root / "out" / "review.json"
    out.unlink(missing_ok=True)
    proc = subprocess.run(
        ["python3", "review.py", "--fixture", str(fixture), "--out", str(out)],
        cwd=root,
        capture_output=True,
        text=True,
        timeout=120,
        env={"PATH": "/usr/bin:/bin", "HOME": "/tmp/empty-home"},
    )
    if proc.returncode != 0 or not out.is_file():
        print("FAIL command", proc.returncode)
        print(proc.stderr[-400:])
        return 1
    review = json.loads(out.read_text())
    if review.get("risk") != packet["expected_risk"]:
        print("FAIL risk", review.get("risk"))
        return 1
    reason = str(review.get("reason", "")).strip()
    if not reason or len(reason) > 240:
        print("FAIL reason shape")
        return 1
    print("PASS", packet.get("server"), packet.get("model_route"))
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Notice the grader does not honor packet["command"] as a shell string. You already published the command shape. Letting a submission choose a shell on your laptop is how a take-home becomes an incident. The packet command is documentation. The argument vector above is the contract.

Hash the fixture on your side, then run the grader against a checkout.

python3 - <<'PY'
import hashlib, pathlib
p = pathlib.Path("fixtures/risky.diff")
print(hashlib.sha256(p.read_bytes()).hexdigest())
PY
python3 grade_replay.py ./candidate-repo

The scrubbed environment is the analogy you should keep. You are checking the sample under your light, not under theirs. A helper that only imports because a key was exported in their terminal dies here. That death is a successful grade of the packet, even when it feels rude the first afternoon you see it.

The 120 second timeout is a gate you chose so a hung socket cannot sit in your calendar until lunch. It is not a performance score, and it is not a claim about any host. If a free server is slow, you rerun once, or you accept the offline backstop. You do not convert jitter into a character reference.

A sample solution that survives an unplugged route

A strong submission is dull. The offline rule is the backstop. A model call is optional garnish.

If you unplug the route and the file still appears with risk set to high, they understood the assignment. If the file appears only when a vendor is awake, they shipped a demo.

#!/usr/bin/env python3
"""Sample solution sketch. The rule, not the model, is the oracle."""
import json, os, sys
from pathlib import Path

def decide(diff_text: str) -> dict:
    risky = (
        "subprocess" in diff_text
        and "shell=True" in diff_text
        and "request" in diff_text
    )
    if risky:
        return {
            "risk": "high",
            "reason": "shell command built from request input",
        }
    return {
        "risk": "low",
        "reason": "no request-fed shell call in the fixture",
    }

def main() -> int:
    args = sys.argv
    fixture = Path(args[args.index("--fixture") + 1])
    out = Path(args[args.index("--out") + 1])
    review = decide(fixture.read_text())
    route = os.environ.get("MODEL_ROUTE", "").strip()
    if route:
        review["model_route_declared"] = route
    out.parent.mkdir(parents=True, exist_ok=True)
    out.write_text(json.dumps(review))
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Tell them a model may disagree with the rule, and the rule wins on this fixture. That sentence saves you from a confident paragraph that marks the diff low because a nearby comment mentioned tests. Fluency is cheap. The oracle is not.

If they declare a model route, they also declare it was not required for the pass. That is the grown-up use of free model access. The free path can draft the reason sentence. It cannot be the only reason the JSON exists.

A free server, when they used one, gets the same treatment. Name it, prove the command there, or write local and move on. A hostname with no replay behind it is a slide.

This heuristic is a toy oracle for one file. Do not describe it as a scanner, and do not hire someone because three in checks impressed you. You are grading whether they can separate a stable check from a live demo. A real review bot needs a wider net than this fixture. Say that in the invite so a clever candidate does not waste the four hours building a framework you will not run.

How you say the score out loud

You do not need a spreadsheet if you say the same four checks every time.

The seal comes first. The fixture hash matches, secrets are not required, and the command shape is the one you published. A packet that says "see the video" scores zero on this check. You already saw the video. It did not clone.

The second run comes next. Exit code zero, out/review.json appears, and risk matches your oracle. You do not grade the reason for wit. You grade that it is one sentence and that it names the shell pattern. A short novel in that field is a smell, not a bonus.

The boundary comes third. model_route is whatever they called, including offline. network_allow stays empty unless you agreed to a host. A free server earns credit only when server matches where the command ran. A mismatch is a fail. A local run is a pass on this check.

Wait, that last pair is five beats. Say it as two breaths instead. Match the field to the place the command ran. Local is allowed. A lie about the host is not.

Restraint comes last. No extra hosts, no socket opened just to log, no pastebin call hiding in review.py. The helper that phones home is a different product from the helper you asked for. You are allowed to stop reading there.

After a pass, keep three artifacts: the packet, the fixture hash, and the review.json your grader wrote. Drop the screen recording as evidence. If a committee asks how you know, you show the command and the PASS line. That argument is smaller than a fourteen-minute video, and you can repeat it when someone challenges the score six weeks later.

Failure modes that show up after the smile

The haunted laptop is the common one. The README is confident. The client builds itself at import time and demands a key.

Your scrubbed environment never reaches decide. Treat import-time network as a fail. A module that cannot be imported without a secret cannot be replayed, no matter how smooth the call sounded.

The trophy server is the next one. The packet names a free host. Nothing in the repo contacts that host, and your argument-vector run is purely local.

They used the server as a slide. If your replay will not hit the host, the honest field is local. Slides do not get points.

The moving fixture shows up when a weak heuristic cannot see shell=True, so they edit your diff and rehash it. Your copy is the source of truth. Hash mismatch is not a conversation. It is a different assignment, and you did not ask for a different assignment.

The model monologue is subtler. The second run exits zero. risk is low. reason is a long page of generic advice about pinning dependencies.

The command worked, and the judgment is wrong. Your expected risk outvotes the prose. Video grading cannot see this failure, because the prose sounds like seniority.

The last failure is yours. You encode a quota, a brand, or a latency cap you have not verified, then fail someone because a free path was slow on a Tuesday. Free access is a constraint you offer, not a benchmark you publish. If the route is down, rerun later or accept the offline backstop. Do not write the outage into their score.

Who should skip this gate

Skip it when the role is a design hire and the work product is a diagram. A replay will not show taste, and pretending it will waste a week. Skip it when you cannot run stranger code in a sandbox. The grader is a footgun on a machine that holds production tokens, even with the argument vector locked.

Skip it when your loop is a forty-minute pairing session and you will never clone the repo. Sending this prompt and then grading the recording teaches people that the packet was theater.

Also skip it if you wanted a leaderboard. One fixture, one expected risk, one replay: that is a gate, not a ranking.

Two candidates who pass are not ordered by this script. They both left a sample you could open. Order them with a different exercise, on a different day, about a different skill.

The line you leave in the invite

Close the invite with one practical sentence, not a pitch. If a free model path and a free server are on offer through MonkeyCode, the candidate may use either, and packet.json has to show which path the replay used. The offer is a constraint, the way a library card is a constraint. It does not grade itself, and it does not expire on a timer you invented in the rubric.

You still read the reason sentence. You still care that they noticed the shell call and not a typo in a comment. The replay only stops you from hiring the recording. The recording was never the colleague who has to debug this helper on a Monday when the route is down.

Send the Scorecard With the Take-Home

Morgan Zhou — Thu, 24 Sep 2026 18:27:15 +0000

You open the packet at 10:40 p.m. The candidate built "an agent." The README has a GIF, a free-server URL, and three adjectives. You click the URL. It times out. The GIF cannot be graded.

That is the failure this take-home is built to prevent. You are not collecting vibes. You are collecting a scorecard that another engineer can replay on Monday without asking the candidate to narrate the demo again.

A take-home for agent work has four pieces, and they travel together. The prompt states the constraint. The rubric states what a pass looks like. A sample harness shows the shape of a valid submission. The failure notes tell the candidate which impressive-looking artifacts you will reject. Leave any one of those out and you will spend the review call reconstructing intent from a chat log.

Here is the prompt you can paste into the packet. It is deliberately narrow. A wide prompt produces a wide excuse.

Take-home prompt (48 hours, async)

Build a small grading harness for a tool-using assistant.
The assistant receives a task, may call one tool, and returns a short answer.
You do not need a clever model. You need a replayable score.

Requirements:
1. Commit fixtures: at least six tasks, each with an expected tool name and an expected answer fragment.
2. Run those fixtures against an endpoint you control.
3. Write scorecard.json. Each row records task_id, expected_tool, observed_tool, answer_ok, and provider_meta taken from the response. If the response has no model id, write null. Do not fill the gap.
4. Include a check command that fails the build when a row claims a quota, a hardware spec, or a latency SLO that is not present in provider_meta.
5. Document how a reviewer re-runs the check. Do not document a benchmark you did not run.

You may use any free model access and, if you have one, a free server so the reviewer can reach the same endpoint. Availability is not a score. A missing field is an honest null, not a reason to invent a name.

That last sentence is the whole point of the exercise. Free access is a convenience for a short review window. It is not evidence. Candidates love to write that they used "the fast model on the free tier" and then attach a score you cannot regenerate. Your packet should make that sentence uncommitable.

If they need a place to try the loop without provisioning a box, MonkeyCode's free model access and free server option are enough to stand an endpoint up for this packet. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Treat that pair of options as availability, not as a catalog. This walkthrough does not name models, quotas, hardware, or a retention window, because those were not supplied as stable facts. If the console shows a model id, the harness copies that id into provider_meta. If it does not, the field stays null.

The rubric should live in code, not in a slide. A reviewer who cannot execute the rubric will renegotiate it on the call, and then two candidates were not judged on the same task. Think of the rubric as a lock on the door. The key is the scorecard. A charming README does not pick the lock.

# UNEXECUTED EXAMPLE. Proposal only. Not a measured run.
from dataclasses import dataclass

@dataclass(frozen=True)
class Rubric:
    min_tasks: int = 6
    require_tool_match: bool = True
    require_answer_fragment: bool = True
    forbid_unsourced_claims: tuple = ("quota", "tokens_granted", "gpu", "p95_ms", "sla")

def row_passes(row: dict, rubric: Rubric) -> tuple[bool, str]:
    if rubric.require_tool_match and row.get("observed_tool") != row.get("expected_tool"):
        return False, "tool_mismatch"
    if rubric.require_answer_fragment and not row.get("answer_ok"):
        return False, "answer_fragment_missing"
    meta = row.get("provider_meta") or {}
    for key in rubric.forbid_unsourced_claims:
        if key in row and key not in meta:
            return False, f"unsourced:{key}"
    return True, "ok"

Read that slowly. The pass condition is boring on purpose. Tool match, answer fragment, and a ban on numbers the provider did not return. You are hiring for judgment under incomplete information. A candidate who adds a leaderboard column without a source has failed, even if the demo looked smooth.

The sample solution is a harness, not a model wrapper with aspirations. Label it as unexecuted when you send it. You are showing shape. You are not claiming a score. The unfinished function is a feature. It stops a reviewer from pasting this sketch into a score spreadsheet and calling it evidence.

# UNEXECUTED EXAMPLE. Sample solution shape for the take-home.
# Replace grade_one() with a real HTTP call. Do not invent provider fields.
import argparse, json, pathlib, sys

def grade_one(task: dict, endpoint: str) -> dict:
    # Proposal: POST {task_id, prompt} and read the JSON body.
    # Copy only keys the body actually contains: tool, answer, provider_meta.
    raise NotImplementedError("wire this to your endpoint")

def annotate(task: dict, observed: dict) -> dict:
    meta = observed.get("provider_meta") or {}
    answer = observed.get("answer") or ""
    row = {
        "task_id": task["id"],
        "expected_tool": task["expected_tool"],
        "observed_tool": observed.get("tool"),
        "answer_ok": task["answer_fragment"] in answer,
        "provider_meta": meta,
    }
    if row["observed_tool"] != row["expected_tool"]:
        row["reason"] = "tool_mismatch"
    elif not row["answer_ok"]:
        row["reason"] = "answer_fragment_missing"
    else:
        row["reason"] = "ok"
    return row

def check(rows: list[dict]) -> int:
    bad = [r for r in rows if r.get("reason") != "ok"]
    pathlib.Path("scorecard.json").write_text(json.dumps(rows, indent=2))
    print(f"rows={len(rows)} failed={len(bad)}")
    return 1 if bad else 0

def main() -> int:
    p = argparse.ArgumentParser()
    p.add_argument("--fixtures", required=True)
    p.add_argument("--endpoint", default="http://127.0.0.1:8787/grade")
    p.add_argument("--check-only", action="store_true")
    args = p.parse_args()
    if args.check_only:
        rows = json.loads(pathlib.Path("scorecard.json").read_text())
        return check(rows)
    tasks = [
        json.loads(line)
        for line in pathlib.Path(args.fixtures).read_text().splitlines()
        if line.strip()
    ]
    rows = []
    for task in tasks:
        try:
            observed = grade_one(task, args.endpoint)
        except NotImplementedError:
            observed = {"tool": None, "answer": "", "provider_meta": {}}
        rows.append(annotate(task, observed))
    return check(rows)

if __name__ == "__main__":
    sys.exit(main())

Pair that harness with fixtures that look like work, not like riddles. Six lines is enough to start. Each line is one contract: an id, a prompt, the tool you expect, and a fragment that must appear in the answer. If you need a seventh case, add a task whose correct behavior is to call no tool and say so. Do not hide that case. Tell the candidate it exists.

{"id":"t1","prompt":"What is the status of invoice 1044?","expected_tool":"lookup_invoice","answer_fragment":"1044"}
{"id":"t2","prompt":"Do not call tools. Reply with the word ready.","expected_tool":null,"answer_fragment":"ready"}

The commands you put in the packet should be copy-pasteable and secret-free. A reviewer should not need a key you forgot to rotate. If the endpoint wants a token, the candidate documents the env var name and you inject it locally. The repo never contains the value.

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python harness.py --fixtures fixtures/tasks.jsonl --endpoint "$EVAL_ENDPOINT"
python harness.py --check-only

If EVAL_ENDPOINT points at a free server, say so in one line of the README and stop. Do not add a hardware guess beside it. The check command is the review. The prose is optional. A green --check-only on the committed scorecard.json means the file is internally consistent. It does not mean the live URL still answers. Run the live path only when you still have the endpoint, and write a second file if the new rows differ. Never overwrite history to make Friday look like Thursday.

Walk the candidate through the failures you will actually reject, in the same voice you use on the call. A GIF with no scorecard is a pitch, not a submission. A scorecard that names a model the response never returned is a fabricated trace. A writeup that quotes a token grant, a GPU class, or a p95 you did not measure is the same failure as a broken test, because the rubric treats unsourced claims as broken rows. A harness that only runs against a laptop process and then calls that "production on the free server" has mixed two environments. Split them. Local is local. The free server is whatever answered the HTTP call, and only that response may fill provider_meta.

There is a softer failure that still wastes your Friday. The candidate returns a perfect score on six toy tasks and a paragraph about how the agent "understood intent." You asked for tool match and an answer fragment. Understanding is not in the rubric. Do not promote it during the review or you have graded a different assignment than the one you sent. That is how two finalists become incomparable. Another common miss is the silent retry. The candidate's script calls the endpoint three times and keeps the flattering row. Your packet should say one attempt per task unless provider_meta records the retry count that the server itself returned. If the server does not return that count, the harness does not invent one.

Clock skew shows up too. A candidate stamps every row with the time on their laptop and then argues the free server was "fast." Latency belongs in the row only when the response, or a timer wrapped around that single call, produced it. A hand-typed 42ms is a story. Stories go in the cover letter. They do not go in scorecard.json.

Limitations belong in the packet, not in a footnote you remember later. This approach does not measure safety, refusal quality, or cost. Six fixtures will not catch a tool that deletes the wrong row in production. A free server can disappear between Thursday and the review, so the scorecard file is the artifact of record and the live URL is a convenience. If your process requires a signed vendor, a data-processing agreement, or a fixed quota, do not use a free option as the system under test. Run the same harness against the endpoint your security review already allowed, and keep the null-filling rule. Free availability can also change without notice. Do not write a hiring policy that assumes this week's console will look the same next quarter.

Who should skip this take-home: interviewers who need a live whiteboard and a shared editor, teams hiring for research taste rather than operational grading, and anyone who will override the rubric when a charismatic README disagrees with it. Also skip it if you cannot spend thirty minutes re-running --check-only. A scorecard you will not execute is just a longer GIF. Skip it too if the role's real risk is data leaving a boundary you have not described. This packet does not teach that boundary. It only teaches you to grade what was written down.

The sample above is a proposal. It has not been executed against a live endpoint in this article, and grade_one is intentionally unfinished so you do not mistake a sketch for a result. Wire it, run the six fixtures, and commit the scorecard the command actually wrote.

If a free model path and a free server are already on the table, point EVAL_ENDPOINT there, copy only the fields the response returns, and leave the rest null. That is the submission. The rest of the pitch can wait for the call.

Grade the Tool Call, Not the Writeup

Morgan Zhou — Wed, 23 Sep 2026 16:07:32 +0000

You unzip the take-home at 9:12. The README would make a staff engineer nod. There is a latency table with two decimal places. One paragraph swears the agent called an API, got a 200, and handled the error path.

You grep the archive for a request body, a tool name, a transcript that is not a paragraph. Nothing. The numbers are round. The timestamps are rounder. You have been handed a book report with a plot about a server.

That is the miss this assignment is built to catch. Not whether a model can sound senior. Whether a tool actually ran with an input you minted, and whether the packet still contains that run.

Flip a coin before you send the prompt

Before you mail anything, you mint a value the candidate cannot have memorized. Sixteen bytes of hex. You keep a copy. They do not get a second one. If their artifact cannot show that this exact string entered a tool and came back transformed, the essay is decoration.

# mint_nonce.py — run once per candidate, on your machine
import secrets
from datetime import datetime, timezone

nonce = secrets.token_hex(16)
issued_at = datetime.now(timezone.utc).isoformat()
print(f"{issued_at} {nonce}")
open("NONCE.txt", "w").write(nonce + "\n")

Think of it as a waiter’s ticket. Anyone can describe a kitchen. Only the ticket that left your hand and came back with grease on it proves an order was cooked. A trace id you did not issue can be invented. A nonce you issued cannot, unless they actually held it.

The prompt you actually hand them

Here is a take-home you can send as-is. Time-box it. Do not ask for architecture theater. The point is a tool call you can replay, not a strategy memo.

You have 45 minutes.

You are given NONCE=<paste the hex here>.
You must not hard-code a fake weather payload.

Implement a single tool named lookup_city that accepts
{ "city": string, "nonce": string }.

The tool must:
1. Reject a city that is empty or only whitespace.
2. Include the nonce on the outbound side as header X-Takehome-Nonce
   if you actually perform HTTP; if you stay in-process, still persist
   that header object next to the call.
3. Return JSON with:
   - echo_nonce: the nonce you received
   - nonce_sha256: SHA-256 hex of (nonce + ":" + city)
   - source: "tool" (literal)
   - city: the city argument

Then run an agent loop (model or a stub you label as a stub) that:
- looks up "Lisbon"
- refuses " "

Submit four files, nothing else:
- lookup_city implementation
- tool_call.json (the exact arguments the loop emitted)
- result.json (the exact tool result)
- timeout.md (one page: what you do if the tool never returns)

If any file is missing, the take-home is incomplete.
Do not submit screenshots.

You are not grading prose. You are grading whether a tool was specified, invoked, and answered with your coin still inside it. The timeout page is the only human writing you should need. Everything else a script can fail in seconds.

A rubric that refuses vibes

Score the packet, not the confidence. The table below is the whole interview loop for this assignment. Four of five gates are mechanical. The fifth is whether they even attempted the empty-city path.

Gate	Pass	Fail
Nonce echo	`echo_nonce` matches the issued hex, byte for byte	Missing, truncated, or a different string
Binding	`nonce_sha256` equals SHA-256 of `nonce + ":" + city` for Lisbon	Hash of nonce alone, hash of city alone, or a made-up digest
Tool shape	One function `lookup_city` with both arguments	A free-form essay that "would call" an API
Refusal	Empty city does not produce a success object	Whitespace still returns a weather story
Transcript	Submitted JSON is valid and matches the implementation	Screenshots, paraphrases, or "the model said it worked"

A proposal grader you can run while the writeup tries to charm you:

# grade_takehome.py — proposal, unexecuted against your files until you run it
import hashlib
import json
import sys

issued = sys.argv[1]
city = "Lisbon"
result = json.load(open("result.json"))
call = json.load(open("tool_call.json"))

expected = hashlib.sha256(f"{issued}:{city}".encode()).hexdigest()

assert call.get("name") == "lookup_city", "wrong or missing tool name"
assert call.get("arguments", {}).get("nonce") == issued, "nonce never entered the call"
assert call.get("arguments", {}).get("city") == city, "city never entered the call"
assert result.get("echo_nonce") == issued, "nonce was not echoed"
assert result.get("nonce_sha256") == expected, "hash is not bound to city"
assert result.get("source") == "tool", "source was not the tool"
assert result.get("city") == city, "city mismatch"
print("automatic gates passed; now open timeout.md yourself")

If they never emitted {"city": " ", "nonce": "..."} in a second transcript, they skipped the path that shows up when an agent gets sloppy with arguments. That is not a style issue. That is the bug you will debug at 2 a.m.

A sample solution you can read in one sitting

This is a sample, not a production agent. It records the call the way the prompt asked, instead of summarizing it after the fact. If you wire a real model later, the only extra job is to dump the raw tool-call message before you execute it. Pretty logs that round-trip through a formatter are how nonces die.

# lookup_city.py
import hashlib

def lookup_city(city: str, nonce: str) -> dict:
    header = {"X-Takehome-Nonce": nonce}
    if not city or not city.strip():
        return {
            "error": "city_empty",
            "source": "tool",
            "echo_nonce": nonce,
            "headers": header,
        }
    digest = hashlib.sha256(f"{nonce}:{city}".encode()).hexdigest()
    return {
        "echo_nonce": nonce,
        "nonce_sha256": digest,
        "source": "tool",
        "city": city,
        "headers": header,
    }

# agent_loop.py — persist the call, then run the tool
import json
from lookup_city import lookup_city

def emit_tool_call(city: str, nonce: str) -> dict:
    # Label: in a real loop this JSON comes from the model.
    # You still write it to disk before lookup_city runs.
    return {
        "name": "lookup_city",
        "arguments": {"city": city, "nonce": nonce},
    }

if __name__ == "__main__":
    nonce = open("NONCE.txt").read().strip()
    call = emit_tool_call("Lisbon", nonce)
    result = lookup_city(**call["arguments"])
    open("tool_call.json", "w").write(json.dumps(call, indent=2))
    open("result.json", "w").write(json.dumps(result, indent=2))
    refused = lookup_city(" ", nonce)
    open("refusal.json", "w").write(json.dumps(refused, indent=2))
    print("wrote tool_call.json, result.json, refusal.json")

A tiny check for the refusal path, so you are not grading that one by eye:

python - <<'PY'
from lookup_city import lookup_city
out = lookup_city(" ", open("NONCE.txt").read().strip())
assert out["error"] == "city_empty"
assert "nonce_sha256" not in out
print("empty city refused")
PY

If you do involve HTTP, keep the nonce in a header you control and echo it in the body. A status code alone is a shrug. Anyone can type 200.

Where a free model and a free server actually help

You can run this loop on a laptop. Plenty of people should. The moment a model enters the loop, local keys and "I will paste the output later" start leaking into the take-home. Candidates stall. Agents invent tool results because the runtime never existed. The writeup gets prettier as the evidence gets thinner.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access and free server option are enough to execute the loop above without standing up paid inference or a box you babysit for a 45-minute assignment. Use that when the point of the exercise is "did a tool run," not "did we negotiate a cloud account." This article does not attach model names, quotas, hardware, or uptime promises; those change, and the rubric should not depend on them. Keep the nonce on your side. The remote box is a kitchen. The ticket still starts in your hand.

If you want a disposable runtime for the same 45-minute packet, that free server option is one place to try the loop without turning the take-home into an infra project.

How this usually breaks

The confident writeup with no tool_call.json is the classic miss. It reads well in a hurry. It cannot echo a coin it never held. You will be tempted to score the prose anyway. Do not.

The second miss is hashing the nonce alone. That proves they saw the string. It does not prove the city argument participated. Bind the two. Otherwise a template can replay last week’s digest and look exact.

The third miss is a screenshot of a chat window. Compression eats hex. You cannot grade pixels. Demand the JSON. If their tooling cannot export it, the tooling is part of the fail.

The fourth miss is executing lookup_city in the writeup and never letting the model, or the labeled stub, emit arguments. That is a unit test of a function you already have. The take-home is about the call. If nothing chose the tool, you learned nothing about the agent.

The fifth miss is leaking the nonce into a public gist and then being surprised when a later packet hashes it perfectly. Mint per candidate. Write the issued value next to the submission in notes you own. Expire it in those notes even if the server will not.

The sixth miss is a timeout page that is another invented latency table. You asked what they would do when the tool hangs. If the answer is "retry until it works" with no cap, no idempotency note, and no human fallback, they will hang you later.

Who should not use this

Do not use a nonce gate as a proxy for whether a human is smart. It measures whether an execution claim is grounded. That is a narrower question, and it is the only one this artifact answers. A brilliant candidate who never dumped the tool call still fails this packet. A weak agent that lucked into the hash still only proved the hash.

Skip it if your hiring process forbids sending unique tokens to candidates, or if you cannot store the issued nonce next to the zip. Skip it if the tool under test must run fully offline with no extra runtime; a free remote server will not make that constraint honest. Skip it if you need a vendor SLA, audit logs that survive a lawsuit, or a promise that a free tier will still be there on the day you interview. This workflow assumes a disposable 45-minute box and a script you own.

You flip the coin. You keep the other side. When the hash comes back bound to Lisbon, you have a take-home. When it does not, you have a story. Grade the first one.

Fetch the Trace Id Before You Hire

Morgan Zhou — Tue, 22 Sep 2026 14:25:11 +0000

You open a take-home at 11 p.m. and the README is a novella. Three screenshots of a chat panel sit under a heading called Demo. The last bubble reads Order created. There is no URL, no request body, no status code. You cannot tell if a model called anything, or if someone typed a happy path into Markdown and went to bed.

That is not a candidate problem. It is an assignment problem. If the role includes models that call APIs, a writeup is a story about a wire you never saw. You need a file you can replay.

The job moved. Most take-home zips did not. They still ask for a feature and a paragraph, the same shape that worked when the work was a CRUD app on localhost. It fails when the interesting part is a tool call. The model must pick a function, fill arguments, hit your HTTP boundary, and react to the status code that comes back. Chat logs lie about that loop. They summarize. They omit the 429. They never show that the Idempotency-Key on the retry did not match the first attempt.

You do not need a bigger rubric. You need the candidate to submit the trace. Think of it like a flight recorder. You do not grade the pilot's essay about the landing. You pull the box and look at airspeed. A JSONL of HTTP request and response is that box. If they cannot produce one, they did not close the loop. They narrated it.

Keep the prompt short enough to read on a phone. Paste something in this shape and label it as the assignment, not as a war story from a team you have not named.

Build a tiny shop stub and a harness that drives it through a model.

The stub exposes:
  POST /orders
    header Idempotency-Key required
    JSON body: {"sku": string, "qty": int}
    201 on first unique key; body includes order_id and trace_id
    409 if the same key arrives with a different body
    200 if the same key arrives with the same body (replay)
  GET /orders/{id}
  GET /_trace/{trace_id}  — the exact request/response the stub stored

The harness:
  - talks to a model over HTTP (base URL and model name from env)
  - exposes one tool: create_order(sku, qty)
  - must actually POST to the stub; do not invent the 201
  - writes replay.jsonl, one object per HTTP hop, including trace_id

Submit:
  1. replay.jsonl
  2. the stub and harness
  3. a base URL where the stub is still reachable for 48 hours

We will replay two rows from your JSONL against your URL.
If they do not match, the take-home is incomplete.

You are not asking them to train a model. You are asking them to prove a round trip. That is closer to production than another todo API, and it is still small enough for a weekend. The trick is to ship a stub that remembers what it saw, then refuse to score anything the stub cannot echo back.

The following stub is a reference solution, not a transcript from a billed service. It keeps orders and traces in memory, stamps X-Trace-Id, and lets you fetch that trace later. Read it as a starting point the candidate may replace, as long as /_trace/{id} still tells the truth.

# stub_shop.py — reference stub for the take-home
from http.server import BaseHTTPRequestHandler, HTTPServer
from urllib.parse import urlparse
import json, uuid, threading

LOCK = threading.Lock()
ORDERS = {}   # order_id -> record
KEYS = {}     # idempotency_key -> order_id
TRACES = {}   # trace_id -> blob

class Shop(BaseHTTPRequestHandler):
    def log_message(self, fmt, *args):
        return

    def _read(self):
        n = int(self.headers.get("Content-Length", 0))
        return json.loads(self.rfile.read(n) or b"{}")

    def _send(self, code, payload, trace_id):
        raw = json.dumps(payload).encode()
        self.send_response(code)
        self.send_header("Content-Type", "application/json")
        self.send_header("X-Trace-Id", trace_id)
        self.send_header("Content-Length", str(len(raw)))
        self.end_headers()
        self.wfile.write(raw)

    def do_POST(self):
        if urlparse(self.path).path != "/orders":
            return self._send(404, {"error": "nope"}, str(uuid.uuid4()))
        key = self.headers.get("Idempotency-Key")
        body = self._read()
        trace_id = str(uuid.uuid4())
        bad = (not key or "sku" not in body or not isinstance(body.get("qty"), int))
        if bad:
            blob = {"path": "/orders", "key": key, "body": body, "status": 400}
            with LOCK:
                TRACES[trace_id] = blob
            return self._send(400, {"error": "bad request", "trace_id": trace_id}, trace_id)
        with LOCK:
            if key in KEYS:
                existing = ORDERS[KEYS[key]]
                status = 200 if existing["body"] == body else 409
                blob = {"path": "/orders", "key": key, "body": body,
                        "status": status, "order_id": existing["order_id"]}
                TRACES[trace_id] = blob
            else:
                order_id = "ord_" + uuid.uuid4().hex[:8]
                ORDERS[order_id] = {"order_id": order_id, "body": body, "key": key}
                KEYS[key] = order_id
                status = 201
                blob = {"path": "/orders", "key": key, "body": body,
                        "status": status, "order_id": order_id}
                TRACES[trace_id] = blob
            order_id = blob.get("order_id")
        payload = {"trace_id": trace_id, "order_id": order_id, "ok": status in (200, 201)}
        if status == 409:
            payload = {"trace_id": trace_id, "error": "idempotency conflict"}
        self._send(status, payload, trace_id)

    def do_GET(self):
        parts = urlparse(self.path).path.strip("/").split("/")
        trace_id = str(uuid.uuid4())
        if len(parts) == 2 and parts[0] == "_trace":
            rec = TRACES.get(parts[1])
            if not rec:
                return self._send(404, {"error": "unknown trace"}, trace_id)
            return self._send(200, rec, trace_id)
        if len(parts) == 2 and parts[0] == "orders":
            rec = ORDERS.get(parts[1])
            if not rec:
                return self._send(404, {"error": "unknown order"}, trace_id)
            return self._send(200, rec, trace_id)
        self._send(404, {"error": "nope"}, trace_id)

if __name__ == "__main__":
    HTTPServer(("0.0.0.0", 8080), Shop).serve_forever()

Start it with python stub_shop.py. Then prove the stub without a model at all. That sanity check belongs in the zip. Candidates who skip it usually invent the JSONL later, and you will feel it when /_trace 404s.

# first create -> 201 and an X-Trace-Id
curl -sS -D - http://127.0.0.1:8080/orders \
  -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: k1' \
  -d '{"sku":"sku_light","qty":2}'

# same key, same body -> 200
curl -sS -D - http://127.0.0.1:8080/orders \
  -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: k1' \
  -d '{"sku":"sku_light","qty":2}'

# same key, different body -> 409
curl -sS -D - http://127.0.0.1:8080/orders \
  -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: k1' \
  -d '{"sku":"sku_light","qty":9}'

Copy X-Trace-Id from the first response and hit /_trace/<that-id>. If that round trip does not work on localhost, it will not work after they deploy. You are teaching them the scoring move before they ever call a model. That is kindness, not a leak. A take-home that hides the grading wire is just a riddle.

The harness is the other half of the sample. It should refuse to write a success row unless the stub returned one. Environment variables keep vendor names out of the zip. Point it at any OpenAI-style chat endpoint. The script below is a labeled reference driver. It is not a claim that it was run against a paid account.

# harness.py — reference driver
import json, os, uuid, urllib.request, urllib.error

STUB = os.environ["STUB_URL"].rstrip("/")
MODEL_URL = os.environ["MODEL_BASE_URL"].rstrip("/")
MODEL = os.environ["MODEL_NAME"]
REPLAY = open("replay.jsonl", "a")

TOOLS = [{
    "type": "function",
    "function": {
        "name": "create_order",
        "description": "Create an order in the shop stub",
        "parameters": {
            "type": "object",
            "properties": {
                "sku": {"type": "string"},
                "qty": {"type": "integer"}
            },
            "required": ["sku", "qty"]
        }
    }
}]

def hop(method, url, body=None, headers=None):
    data = None if body is None else json.dumps(body).encode()
    hdrs = {"Content-Type": "application/json"}
    if headers:
        hdrs.update(headers)
    req = urllib.request.Request(url, data=data, method=method, headers=hdrs)
    try:
        with urllib.request.urlopen(req, timeout=20) as resp:
            raw = resp.read()
            rec = {
                "url": url, "method": method, "status": resp.status,
                "req": body, "resp": json.loads(raw.decode() or "{}"),
                "trace_id": resp.headers.get("X-Trace-Id"),
            }
    except urllib.error.HTTPError as e:
        raw = e.read()
        rec = {
            "url": url, "method": method, "status": e.code,
            "req": body, "resp": json.loads(raw.decode() or "{}"),
            "trace_id": e.headers.get("X-Trace-Id"),
        }
    REPLAY.write(json.dumps(rec) + "\n")
    REPLAY.flush()
    return rec

def create_order(sku, qty):
    key = "idem_" + uuid.uuid4().hex[:12]
    return hop(
        "POST", f"{STUB}/orders",
        {"sku": sku, "qty": qty},
        {"Idempotency-Key": key},
    )

def call_model(messages):
    payload = {"model": MODEL, "messages": messages, "tools": TOOLS}
    rec = hop("POST", f"{MODEL_URL}/chat/completions", payload)
    return rec["resp"]

if __name__ == "__main__":
    messages = [
        {"role": "system", "content": "Create shop orders via tools. Never pretend the order exists."},
        {"role": "user", "content": "Order 2 units of sku_light."},
    ]
    first = call_model(messages)
    msg = first["choices"][0]["message"]
    messages.append(msg)
    for tool_call in msg.get("tool_calls") or []:
        args = json.loads(tool_call["function"]["arguments"])
        result = create_order(**args)
        messages.append({
            "role": "tool",
            "tool_call_id": tool_call["id"],
            "content": json.dumps(result["resp"]),
        })
    final = call_model(messages)
    print(json.dumps({"final": final, "replay": "replay.jsonl"}, indent=2))

The important line is not the model call. It is REPLAY.write. If the model hallucinates an order_id, that id will not show up under /_trace on the stub. You will notice. That is the whole point of making the server the source of truth instead of the chat transcript.

Sit down with two terminals, not with their README. Pull two trace_id values from replay.jsonl. Curl their still-live stub. Compare JSON. You are grading three questions only. Did the model emit a tool call, or did they paste a 201 into the log by hand? Did the stub enforce idempotency, or is every POST a new order? Do the bytes on the wire match the file they zipped?

A writeup can still exist. It just cannot replace those three checks. If you want a paper trail for the hiring committee, record the curls you ran. That recording is yours. Their screenshots are not. A candidate who argues that the chat screenshot is easier to read is telling you they optimized for looking finished. You are hiring for a closed loop.

The collapses rhyme once you have scored a few of these. The JSONL contains only the model chat and no stub hops, which means they logged the narrator and not the tool. The stub still binds 127.0.0.1 and they forgot a public URL, so your Monday replay dies in DNS. They log status codes but drop Idempotency-Key, so you cannot tell a 200 replay from a lucky second insert. They let the model invent order_id when the stub is down, and the harness never fails closed. They rotate a fresh URL after you start scoring. They submit a 400-line architecture novel and a three-line trace.

Each of those is a signal, not a personality test. Missing keys is a product bug. A dead URL is an ops miss, and you should say so in the prompt if process restarts wipe in-memory maps. A hallucinated success row is the one you cannot hire for a tool-calling role. That is how phantom orders reach finance. A beautiful stub that never returns 409 is also a fail, because they deleted the interesting branch to make the demo pretty.

You should not require a paid API key to finish this assignment. The candidate is already giving you a weekend. If the harness reads MODEL_BASE_URL and the stub binds 0.0.0.0, they can point both at something that costs them nothing and still leave you a URL to curl.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option. That pair is enough to host the stub at a public URL and to drive the harness without pasting a vendor key into the zip. Use it as plumbing. It does not grade the trace for you, it does not promise the model will call the tool on the first try, and it does not replace the two curl checks you run on Monday. If you already have a lab box and an internal model gateway, skip the product. The assignment does not depend on it.

Do not ship this zip for a role that never touches HTTP. You will filter for the wrong muscle. Do not use it if your legal team forbids any model in take-homes, because the harness is explicit about calling one. Do not use it as live onsite theater. The value is the URL that still answers after the candidate has slept, not the performance of watching someone type.

Treat the stub as a toy. In-memory maps die on restart, so a free process that recycles will empty /_trace. Say that in the prompt. Ask for the JSONL anyway, and treat a 404 on replay as a deployment note unless the file itself is empty or internally inconsistent. Do not claim the method measures reasoning. It measures whether a tool call happened and whether the candidate can keep a server honest long enough for you to look.

You will still read the code. A perfect trace with the 409 path commented out is a no. A messy trace with an honest conflict and a retry is a yes. The file is evidence. It is not the hire. When you close the laptop this time, you want two matching JSON objects, not a screenshot of a chat bubble that says Order created. If they cannot replay it, you did not grade it. You read a story.

Hand Them a Wrong Answer On Purpose

Morgan Zhou — Mon, 21 Sep 2026 12:36:05 +0000

You unzip the take-home at 9:40 p.m. and the README is a dare. Build a small review service. Return JSON. Be production-minded. There is no example of a bad response, no score sheet, and no note about what happens if your usual model tab is out of credits. You guess. Reviewers will guess later, in a different room, with a different model open.

That guessing is the bug. A take-home that only ships a prompt is a Rorschach test. Two interviewers can read the same repo and argue about taste while the candidate's actual work sits in a folder named final_final. You need a packet that calibrates both sides before anyone types a score.

The packet is four files that travel together: the prompt the candidate sees, a machine-checkable rubric, a sample solution that is wrong on purpose, and a short catalog of how that sample fails. You are not hiding the answer key. You are publishing a known-bad baseline so a free model, a paid model, and a tired human reviewer are staring at the same miss.

This is not a personality test. It is a shared measuring stick. If the candidate's service cannot beat the broken sample on the published checks, you are not debating elegance. You are looking at a miss that the zip file already named.

The Sunday-night prompt

Keep the prompt short enough to read on a phone. Long prompts become fan fiction for whoever has the largest context window. You want a job a person can finish on a free model and a free box, because the interview should not be a proxy for who already pays for a coding assistant.

# takehome/PROMPT.md

Build a local HTTP service on port 8080.

POST /review
Content-Type: application/json

{
  "diff": "string",
  "tests_passed": 0,
  "tests_failed": 0,
  "secrets_hit": false
}

Respond with:
{
  "score": 0-100,
  "verdict": "reject" | "revise" | "pass",
  "reasons": ["short, specific strings"],
  "beats_sample": true or false
}

Rules:
- If tests_failed > 0, verdict cannot be pass.
- If secrets_hit is true, score cannot exceed 20 and verdict must be reject.
- reasons must mention a concrete signal from the payload, not a vibe.
- beats_sample is true only when your score is lower than the known-bad sample
  on the same payload (the sample always returns 100).

Do not call a paid API from the service. Stdlib only. Ship grade_receipt.json
that repeats one request and one response you actually ran.

Notice what the prompt refuses to do. It does not ask for Kubernetes. It does not ask for a login with a vendor. It does not ask the candidate to impress you with a dashboard. The work is a contract: a port, a shape, a couple of invariants, and a receipt.

A rubric that can fail a bot

Write the rubric as code the reviewer runs, not as adjectives in a spreadsheet. Adjectives drift. A function either returns 1 or it does not.

# takehome/grade.py
from __future__ import annotations

import json
import urllib.request
from dataclasses import dataclass

SAMPLE_SCORE = 100  # the known-bad sample always claims perfection

CASES = [
    {
        "name": "red_tests_cannot_pass",
        "payload": {
            "diff": "def add(a, b):\n    return a-b\n",
            "tests_passed": 1,
            "tests_failed": 2,
            "secrets_hit": False,
        },
        "forbid_verdict": "pass",
    },
    {
        "name": "secret_caps_score",
        "payload": {
            "diff": "AWS_SECRET_ACCESS_KEY=AKIA...\n",
            "tests_passed": 8,
            "tests_failed": 0,
            "secrets_hit": True,
        },
        "max_score": 20,
        "require_verdict": "reject",
    },
]

@dataclass
class Row:
    name: str
    ok: bool
    detail: str

def post(payload: dict) -> dict:
    req = urllib.request.Request(
        "http://127.0.0.1:8080/review",
        data=json.dumps(payload).encode(),
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=3) as resp:
        return json.loads(resp.read().decode())

def check(case: dict, body: dict) -> Row:
    name = case["name"]
    if not isinstance(body.get("reasons"), list) or not body["reasons"]:
        return Row(name, False, "reasons missing")
    if case.get("forbid_verdict") and body.get("verdict") == case["forbid_verdict"]:
        return Row(name, False, "verdict ignored test failures")
    if case.get("require_verdict") and body.get("verdict") != case["require_verdict"]:
        return Row(name, False, "secret path did not reject")
    if body.get("score", 101) > case.get("max_score", 100):
        return Row(name, False, "score above cap")
    if body.get("beats_sample") is not True:
        return Row(name, False, "did not beat the known-bad sample")
    if body.get("score", 0) >= SAMPLE_SCORE:
        return Row(name, False, "score tied the broken sample")
    blob = " ".join(str(r) for r in body["reasons"]).lower()
    if name == "secret_caps_score" and "secret" not in blob:
        return Row(name, False, "reasons never mentioned the secret")
    return Row(name, True, "ok")

if __name__ == "__main__":
    rows = []
    for case in CASES:
        try:
            rows.append(check(case, post(case["payload"])))
        except Exception as exc:
            rows.append(Row(case["name"], False, f"transport: {exc}"))
    print(json.dumps([row.__dict__ for row in rows], indent=2))
    raise SystemExit(0 if all(r.ok for r in rows) else 1)

Run it the same way the candidate will run it. Same host. Same timeout. Same payload bytes. If you score a recording instead of a live process, you are grading theater.

python3 takehome/grade.py; echo exit:$?

The sample that always says 100

This is the piece most packets skip. You include a server that looks finished and is wrong. Candidates may read it. Reviewers must run it. The point is not cruelty. The point is a shared scar: everyone has seen the same overconfident JSON.

# takehome/sample_bad.py
from http.server import BaseHTTPRequestHandler, HTTPServer
import json

class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        length = int(self.headers.get("Content-Length", "0"))
        _ = self.rfile.read(length)
        body = {
            "score": 100,
            "verdict": "pass",
            "reasons": ["looks good to me"],
            "beats_sample": True,
        }
        raw = json.dumps(body).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(raw)))
        self.end_headers()
        self.wfile.write(raw)

    def log_message(self, fmt, *args):
        return

if __name__ == "__main__":
    HTTPServer(("127.0.0.1", 8080), Handler).serve_forever()

Start it, then hit the grader in another terminal. Watch both cases drop. That red output is now part of the assignment, not a surprise you spring in the debrief.

python3 takehome/sample_bad.py &
python3 takehome/grade.py
# expect exit 1, secret_caps_score and red_tests_cannot_pass both false
kill %1

A candidate who pastes this file and changes the port has not completed the take-home. They have cloned the scar. Your rubric already says so.

A direction that can pass

Label this as a direction, not as the one true house style. It is a proposal you keep in the interviewer folder so a new reviewer does not invent a third scale at 11 p.m.

# takehome/sample_ok.py  (proposal: interviewer copy, not a hidden gold file)
from http.server import BaseHTTPRequestHandler, HTTPServer
import json

class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        n = int(self.headers.get("Content-Length", "0"))
        payload = json.loads(self.rfile.read(n).decode() or "{}")
        failed = int(payload.get("tests_failed") or 0)
        secret = bool(payload.get("secrets_hit"))
        reasons = []
        score = 80
        verdict = "pass"

        if failed:
            score = min(score, 40)
            verdict = "revise"
            reasons.append(f"tests_failed={failed} so pass is illegal")
        if secret:
            score = min(score, 20)
            verdict = "reject"
            reasons.append("secrets_hit=true; score capped at 20")
        if not reasons:
            reasons.append("no failed tests and no secret flag")

        body = {
            "score": score,
            "verdict": verdict,
            "reasons": reasons,
            "beats_sample": score < 100,
        }
        raw = json.dumps(body).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(raw)))
        self.end_headers()
        self.wfile.write(raw)

    def log_message(self, fmt, *args):
        return

if __name__ == "__main__":
    HTTPServer(("127.0.0.1", 8080), Handler).serve_forever()

The interesting line is beats_sample. You forced the candidate to compare themselves to a published failure, not to an imaginary senior engineer. Models are good at emitting a service that “looks like a review API.” They are sloppier when the prompt says they must undercut a specific broken number.

That sloppiness is useful in 2026, when a take-home can look complete because a coding model emitted a tree of files. Polished folders are cheap. Beating a named bad baseline is still work.

Where a free model and a free server actually help

You should run the packet the way a candidate without a corporate card would run it. If the exercise dies without a paid key, you did not measure engineering. You measured billing.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project with free model access and a free server option, which is enough to draft sample_ok.py, keep the service on a public box for the review window, and re-run grade.py without asking the candidate to paste a personal token into your zip. Use it if it fits. Use any other free endpoint if it does not. The packet does not care about the brand on the model card. It cares that the HTTP contract holds when the model is not the one you use at work.

A practical loop looks like this. You put the four files in a repo. The candidate boots the broken sample, watches grade.py fail, then writes their own handler. They deploy that handler on a free server you both can hit, or they keep it on localhost and send grade_receipt.json plus the command they ran. You replay the receipt against the same cases. If the replay and the live port disagree, you stop scoring features and you talk about the receipt.

{
  "command": "python3 takehome/grade.py",
  "endpoint": "http://127.0.0.1:8080/review",
  "cases": [
    {"name": "red_tests_cannot_pass", "ok": true, "detail": "ok"},
    {"name": "secret_caps_score", "ok": true, "detail": "ok"}
  ]
}

Ask them to generate that file from the grader, not by hand. A handwritten receipt is another place a model can lie with perfect grammar.

Failure modes you will actually see

The first miss is cosmetic honesty. The service returns verdict: "revise" and still sets score to 100. Humans call that hedging. The grader calls it a tie with the broken sample. You do not need a debate.

The second miss is reason-shaped smoke. reasons contains "needs more tests" while the payload already told them two tests failed. The model is commenting on the universe. The rubric asked it to quote the payload.

The third miss is the secret path with a polite shrug. secrets_hit is true, verdict is reject, and score is 88 because the diff also “had good structure.” That is how a take-home turns into a style contest. The cap exists so you do not have that contest.

The fourth miss is transport theater. The candidate’s README shows a curl against a tunnel that no longer exists. grade.py throws connection refused. If they shipped a receipt from a live run, you can still talk. If they shipped screenshots, you are done.

The fifth miss is the packet eating itself. Someone “improves” grade.py to be nicer. Now the known-bad sample passes. You no longer have a calibration set. Freeze the grader in the zip. Put proposed changes in a separate pull request after the hire, not during it.

Limits, and who should not use this

This packet measures a small contract under a weak model and a short clock. It does not measure system design for a multi-region billing platform. If you need that, this is the wrong instrument.

Do not use it for roles where a take-home is already an unpaid weekend. The zip is small on purpose. If your legal team cannot accept candidate code, stop at the design conversation and do not collect a server.

Do not use it if you will not run sample_bad.py yourself. A wrong answer on file only calibrates people who press run. And do not bolt this onto a task that needs a GPU, a private dataset, or production credentials. A free server is not a substitute for a staging account you actually control.

The approach also fails if you secretly re-score with a private rubric. Candidates will notice the public checks were a decoy. Reviewers will go back to arguing about taste. You will have spent a Sunday night to recreate the original mess, only now with more JSON.

If you want one change this week, stop mailing a prompt by itself. Mail a prompt, a grader, and a solution that fails in a boring, documented way. Then let the candidate beat that failure on whatever free model and free server they can actually reach. You will still disagree about hiring. You will disagree about the work, not about who had spare tokens on a Sunday.

Swap the Model Before You Score

Morgan Zhou — Sun, 20 Sep 2026 10:44:11 +0000

You promised a hire/no-hire by lunch. The zip is tidy, the GIF is flattering, and the README's first instruction is export OPENAI_API_KEY. You do not have that key in the interview budget. You also do not have the model they used at 1 a.m.

The recording is a souvenir. It is not a score.

That gap is the assignment. A candidate can look sharp on the model already sitting in their shell. You need to know whether the work still grades when you swap the model and the server under it. Models move. Keys expire. A take-home glued to one vendor is already stale when you unzip it.

Think of it like grading a cooking test after you change the oven. If the dish only works in their kitchen, you learned about their kitchen. You did not learn about their method. The public argument around evals keeps circling the same crack: the tests age faster than the systems they claim to measure. Your take-home should assume the swap, not apologize for it.

The assignment you actually send

The block below is a proposed take-home, not a war story from a named hiring loop. You freeze three things: the prompt set, the score schema, and the rule that the reviewer will change MODEL_NAME. The candidate ships a harness. You point that harness at a different base URL and you expect a score file, not a speech.

Keep PROMPT.md boring. Boring is load-bearing.

# Take-home: model-swap eval harness

Build a CLI that reads cases.json, calls a chat model through
MODEL_BASE_URL and MODEL_NAME, and writes score.json.

Rules:
- HTTP + JSON only. Do not import a vendor SDK by name.
- Temperature must be 0. The rubric grades structure, not vibes.
- score.json must validate against schema/score.schema.json.
- The same command must work after the reviewer changes MODEL_NAME.

You will be graded after we swap the model. If the zip only runs on
the model you used while writing it, the score is zero.

That last sentence is the job. Everything else is plumbing. If you bury the swap in a footnote, they will optimize for the GIF instead.

Put the rubric where a command can read it

A rubric that lives in a doc is a speech. A rubric that lives in rubric.yaml is an input. You want the second one. The harness should fail closed when a required field is missing, the same way a compiler fails closed when a type is missing.

# rubric.yaml
version: 1
max_score: 12
checks:
  - id: schema_valid
    points: 3
    how: score.json matches schema/score.schema.json
  - id: vendor_lock
    points: 3
    how: no vendor SDK import, no hardcoded host
  - id: swap_replay
    points: 4
    how: same cases.json, different MODEL_NAME, both runs emit scores
  - id: freeze
    points: 2
    how: temperature 0 and cases.json untouched

You do not need twelve dimensions. You need four that a stranger can re-run from disk. If a check cannot be decided without watching them talk, it does not belong in this take-home.

A sample harness worth pasting

This sample is labeled on purpose: it is an unexecuted example, not a benchmark and not a latency claim. It talks to whatever chat-completions endpoint you export. It does not know the product name of the model. That ignorance is the feature.

# eval.py
import json, os, sys, urllib.request
from pathlib import Path

BASE = os.environ["MODEL_BASE_URL"].rstrip("/")
NAME = os.environ["MODEL_NAME"]

def chat(prompt: str) -> str:
    body = json.dumps({
        "model": NAME,
        "temperature": 0,
        "messages": [{"role": "user", "content": prompt}],
    }).encode()
    req = urllib.request.Request(
        f"{BASE}/v1/chat/completions",
        data=body,
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=60) as resp:
        data = json.loads(resp.read().decode())
    return data["choices"][0]["message"]["content"]

def grade_reply(reply: str, case: dict) -> dict:
    expect = case["must_contain"]
    hits = [s for s in expect if s.lower() in reply.lower()]
    return {
        "id": case["id"],
        "pass": len(hits) == len(expect),
        "hits": hits,
        "missing": [s for s in expect if s not in hits],
    }

def main(cases_path: str, out_path: str) -> None:
    cases = json.loads(Path(cases_path).read_text())
    rows = [grade_reply(chat(c["prompt"]), c) for c in cases]
    passed = sum(1 for r in rows if r["pass"])
    Path(out_path).write_text(json.dumps({
        "model": NAME,
        "base_url": BASE,
        "passed": passed,
        "total": len(rows),
        "rows": rows,
    }, indent=2))

if __name__ == "__main__":
    main(sys.argv[1], sys.argv[2])

A reviewer should be able to run two commands and get two score files. Same cases. Different MODEL_NAME. If either command dies because a SDK is missing, they failed the lock check before you read a single answer.

export MODEL_BASE_URL=http://127.0.0.1:8080
export MODEL_NAME=reviewer-a
python eval.py cases.json score-a.json

export MODEL_NAME=reviewer-b
python eval.py cases.json score-b.json

python -m json.tool score-a.json >/dev/null
python -m json.tool score-b.json >/dev/null

Those hosts are placeholders. You will substitute the endpoint you actually review against. The shape of the command does not change. That is the whole trick.

Add a lock test so you are not grepping imports by hand at 11:50.

# test_no_vendor_lock.py
from pathlib import Path
import ast

BANNED = {"openai", "anthropic", "google.generativeai"}

def test_no_vendor_sdk():
    tree = ast.parse(Path("eval.py").read_text())
    names = []
    for node in ast.walk(tree):
        if isinstance(node, ast.Import):
            names += [a.name.split(".")[0] for a in node.names]
        if isinstance(node, ast.ImportFrom) and node.module:
            names.append(node.module.split(".")[0])
    locked = BANNED.intersection(names)
    assert not locked, f"vendor SDK imported: {locked}"

def test_base_url_not_hardcoded():
    src = Path("eval.py").read_text()
    assert "api.openai.com" not in src
    assert "MODEL_BASE_URL" in src

Run pytest test_no_vendor_lock.py. If it fails, you can stop. The rest of the rubric is commentary on a zip that already confessed.

How the four checks feel in a real review

Schema valid is the easy one. Either score.json opens or it does not. Treat a pretty Markdown report with no JSON as a miss. You asked for a file a script can read. They sent you an essay.

Vendor lock shows up as a missing wheel. They wrote from openai import OpenAI because that is what their weekend tutorial used. The import is a confession. It says the take-home was built around a brand, not around a contract.

Swap replay is where most zips die. The first model returns a short JSON-ish blob and their grader looks for a key. The second model wraps the same answer in a sentence, and the grader panics. That panic is useful. It tells you they graded a dialect, not a result. A harness that only understands one model's manners is a client for a pet, not a tool.

Freeze is the quiet check. If cases.json was edited to match answers they already saw, you are not measuring a model. You are measuring a spoiler. Hash the cases file in the zip you sent and compare. If the hash moved, the candidate moved the test.

Where a free model and a free server actually help

You still need a place to point MODEL_BASE_URL that does not start an invoice during review. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project with free model access and a free server option, which is enough to replay a frozen prompt set and write score.json twice. That is the entire use. It does not make the candidate faster in production. It does not replace a staging cluster. It lets you finish the swap without asking finance for a key.

If the zip cannot tolerate that swap, do not argue with the GIF. The GIF was never the artifact.

Failure modes you will see by Thursday

The SDK-shaped hole is only the first. The second is scoring prose. They ask the model to "explain the bug," then they grep for the word race. A model that says "data race" passes. A model that says "concurrent mutation of the same counter" fails. Same diagnosis, different costume. Make the case require a stable token, like the test name or an error code, not an adjective.

The third is temperature as personality. They leave temperature at 0.7 because the demo "felt alive." Alive is not a rubric. Variance is a bug when you are trying to compare two models on the same cases.

The fourth is the notebook that only runs if you click top to bottom while logged into their cloud. A reviewer is not their laptop. If the path to score.json is a ritual, the take-home is a performance.

The fifth is silent truncation. The harness catches urlopen errors and writes an empty rows array with passed: 0. That looks like a model failure. It is a network failure wearing a score. Fail loud. Print the HTTP status. Exit non-zero. Do not launder transport errors into "the model is dumb."

What this does not measure

This take-home will not tell you if they can sit in an incident channel. It will not tell you if they can read a profiler. It will not tell you whether they are kind in code review. If the role is "make our one vendor eval suite two percent better," a model-swap assignment is the wrong oven. Give them that vendor and grade that vendor.

Free model access is also not an SLO. Endpoints change. Rate limits exist even when nobody quotes a number here. Do not write the job post as if a review-time server is a production dependency. Do not treat a candidate's latency screenshot as evidence unless you measured it on hardware you control. This article does not include those numbers, because they were not measured here.

Skip the pattern for screens where you only need a short coding signal. Fizzbuzz does not need a model swap. Agent work that will outlive one vendor does.

Ship the swap instructions with the prompt

Put PROMPT.md, rubric.yaml, cases.json, and the two-command replay in the same zip you send. Tell them, in one sentence, that you will change MODEL_NAME before you score. People optimize for the test they can see. If they cannot see the swap, they will hardcode the weekend.

When score-a.json and score-b.json both exist, you are grading a method. When only the GIF exists, you are grading a subscription. Swap the model. Then read the zip.

Score the Client They Point at Your Server

Morgan Zhou — Sat, 19 Sep 2026 09:00:33 +0000

You get the email at 4:07 p.m. The candidate is polite. They say the take-home looked fair until their personal key died in the second hour, and they did not want to put a card on a weekend assignment.

You already know how this ends. They stall, or they finish on a coworker’s editor seat, or they paste a local demo that never left their laptop. You are not grading engineering. You are grading who could afford a model that afternoon.

That is the quiet leak in a lot of “AI-native” take-homes. The prompt lives in your zip. The runtime lives in their wallet. When the two disagree, the pretty transcript wins, and you never see the client they would have shipped on your stack.

So rewrite the assignment until the runtime is something you can name in the README. Not “use any LLM.” A base URL. A free model path. A server you can hand every candidate without asking them to pay. Then grade the client they pointed at that box.

What you are actually hiring for

The argument around AI coding is loud this month: fluent output is not engineering. Fine. You still have to test something on a Tuesday. The useful test is not whether they can summon a fluent agent on a paid laptop. The useful test is whether they can write a small, boring client that talks to a provided endpoint, keeps secrets out of the log, and fails in a way a reviewer can replay.

Think of it like a driving exam that hands everyone the same car. You do not score the stereo. You score whether they signal, brake, and leave the lot without taking the curb with them.

A practical place to pin that car is a hosted coding environment that already offers free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project. This article will not invent model names, token ceilings, or hardware claims. The only facts that matter for the take-home are the two you can put in a README without lying: candidates can reach a free model path, and they can use a free server instead of renting one at home.

If that pair is in the zip, the assignment stops being “impress me with your local agent” and becomes “honor this contract.”

The prompt you put in the zip

Keep the story small. You are not asking them to build a platform. You are asking them to ship a client that cannot quietly wander off to another host.

Paste something like this. Label it as the take-home, not as a production spec.

TAKE-HOME: bound-client (90 minutes, no paid key)

You are given:
- TASK.md — a failing unit test and a short product note
- a base URL in .env.example (do not hardcode)
- a requirement: all model calls go to that URL
- a requirement: write receipt.json after every run

Build a CLI `bound-client` that:
1. Reads TASK.md
2. Sends one JSON request to $BOUND_BASE_URL/v1/complete
3. Writes ./out/patch.diff if the model returns a unified diff
4. Writes ./out/receipt.json with timestamp, request_id,
   endpoint host, no secrets, and exit_reason
5. Exits 2 if the server is unreachable, 3 if the response
   is not a diff, 0 on a well-formed receipt

You may not:
- call any other model host
- print Authorization headers
- require a paid cloud account

Submit: client source, receipt.json from a run against the
provided server, and a 15-line notes.md on one failure you forced.

Notice what is missing. There is no “build an agent framework.” There is no branded model nickname. The candidate who already has a gold local setup does not get a head start, because that gold setup is the wrong host.

Wire the example env to the free server you actually intend to grade.

# .env.example — ship this, never a live secret
BOUND_BASE_URL=https://YOUR_FREE_SERVER_HOST
BOUND_TOKEN=

If you are using MonkeyCode’s free server option for the cohort, put that host in .env.example and nowhere else. Candidates who “accidentally” point at a personal gateway fail the contract even if the patch looks clever.

A sample solution you keep in the answer key

Do not ship this in the candidate zip. Keep it for calibration so two reviewers score the same way. The sample is intentionally dull. Dull is the point.

#!/usr/bin/env python3
"""bound-client: one request, one receipt, no leaked headers."""
from __future__ import annotations

import json
import os
import sys
import time
import urllib.error
import urllib.request
from pathlib import Path
from urllib.parse import urlparse

OUT = Path("out")
RECEIPT = OUT / "receipt.json"
PATCH = OUT / "patch.diff"


def redact(url: str) -> str:
    parsed = urlparse(url)
    return parsed.netloc or "missing-host"


def write_receipt(**payload: object) -> None:
    OUT.mkdir(exist_ok=True)
    RECEIPT.write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8")


def main() -> int:
    base = os.environ.get("BOUND_BASE_URL", "").rstrip("/")
    token = os.environ.get("BOUND_TOKEN", "")
    task = Path("TASK.md").read_text(encoding="utf-8")
    host = redact(base)
    started = time.time()

    if not base:
        write_receipt(endpoint_host="missing", exit_reason="no_base_url")
        return 2

    body = json.dumps({"task": task, "format": "unified_diff"}).encode("utf-8")
    req = urllib.request.Request(
        f"{base}/v1/complete",
        data=body,
        method="POST",
        headers={"Content-Type": "application/json"},
    )
    if token:
        req.add_header("Authorization", f"Bearer {token}")

    try:
        with urllib.request.urlopen(req, timeout=30) as resp:
            raw = resp.read().decode("utf-8")
            request_id = resp.headers.get("X-Request-Id", "none")
    except urllib.error.URLError as exc:
        write_receipt(
            endpoint_host=host,
            request_id="none",
            elapsed_ms=int((time.time() - started) * 1000),
            exit_reason=f"unreachable:{type(exc).__name__}",
        )
        return 2

    text = raw.strip()
    ok_diff = text.startswith("diff --git") or text.startswith("--- ")
    write_receipt(
        endpoint_host=host,
        request_id=request_id,
        elapsed_ms=int((time.time() - started) * 1000),
        exit_reason="ok" if ok_diff else "not_a_diff",
        response_bytes=len(raw),
    )
    if not ok_diff:
        return 3
    PATCH.write_text(text + "\n", encoding="utf-8")
    return 0


if __name__ == "__main__":
    sys.exit(main())

A candidate who wraps five agent libraries around this has extra surface you did not ask for. A candidate who prints the bearer token in debug mode failed a security check you can defend in a debrief.

Force one failure in your own dry run so you know the exits work before anyone else touches the zip.

BOUND_BASE_URL=http://127.0.0.1:1 python bound_client.py; echo $?
# expect 2, and a receipt whose exit_reason starts with unreachable

Then run the happy path against the free server you named. Keep that receipt next to the rubric. Reviewers compare hosts, not vibes.

You can add a tiny assertion so two interviewers do not argue about JSON shape.

# tests/test_receipt_contract.py — reviewer side only
import json
from pathlib import Path

def test_receipt_has_host_and_reason():
    data = json.loads(Path("out/receipt.json").read_text())
    assert data["endpoint_host"]  # must be your free server's host
    assert data["exit_reason"] in {"ok", "not_a_diff"} or data["exit_reason"].startswith("unreachable")
    assert "Bearer" not in json.dumps(data)

Run it the same way you would run a candidate packet.

python -m pytest tests/test_receipt_contract.py -q

If that test is red, you are not looking at a take-home. You are looking at a screenshot of one.

How you score it

Write the rubric as behavior, not adjectives. “Clean code” is how you argue in a hallway. “Receipt host matches .env.example” is how you end the argument.

Gate A is contract: the only host in receipt.json is the one you provided. Gate B is hygiene: no secrets in stdout, stderr, or the receipt. Gate C is failure shape: killing the server produces exit 2 and a receipt, not a stack trace dumped on the candidate’s desktop. Gate D is the patch: if they submit a diff, it must come from that run, not from a weekend they spent on another model.

A candidate can write an ugly client and still pass A through C. That is a hire signal for this role. A candidate can write a gorgeous agent loop that never called your server. That is a no, even if the unit test goes green.

Failure modes you will actually see

Endpoint laundering shows up first. They run the real work at home, then replay a stub against your URL so the receipt looks right. You catch that when elapsed_ms is a joke, or request_id is missing, or notes.md describes a model you never offered.

Log diarrhea is faster to fail. They set debug to true and the token lands in the zip. Automatic no. Do not debate it. The take-home said not to print Authorization.

Framework gravity wastes the clock. They spend ninety minutes wiring an agent toolkit because that is what their blog diet rewards, and they never produce receipt.json. You asked for a client. They built a parade.

Silent success is the one that fools tired reviewers. The server returns an error, they catch the exception, and they write a fake diff by hand so the take-home “passes.” You catch that because you control the free server logs. If your box never saw them, they were not on your box.

That last one is the whole reason to use a free server you can actually look at. A take-home that runs only on their GPU is a magic show. A take-home that has to appear in your access log is a work sample.

Who should not use this

Skip this design if the role never talks to a model. Skip it if you cannot send candidate traffic to a third-party host, even a free one; in that case you need an internal box, and the prompt stays the same but the URL changes. Skip it for staff-level architecture interviews, where the interesting failure is organizational, not a missing receipt. Skip it if you will not read receipt.json. A rubric you do not apply is just another PDF in the zip.

Also skip it if you need this assignment to prove which model is “best.” It will not. Free model access is enough to finish the prompt. It is not a benchmark. Anyone who turns this take-home into a leaderboard is grading the wrong thing.

What you change in the README

Change one line in your current take-home: name the server. Point .env.example at a free model path and a free server you are willing to watch. Keep the sample client in the answer key, not in the candidate folder. When the packet comes back, read the receipt before you read the diff.

If you want a free model path and a free server to pin that README to, MonkeyCode is one option that fits this contract. Use it as the box you hand every candidate, then score the client they pointed at it.

Grade the Ways It Breaks

Morgan Zhou — Fri, 18 Sep 2026 07:02:12 +0000

You unzip the take-home at 11:14 p.m. because the loop closes in the morning. The README is warm. The tests are green. A terminal recording even shows an agent typing like it has a pulse.

Then you hunt for the autopsy. There is no file that names how the thing lies. There is no sample you can rerun on a machine that is not the candidate's laptop. There is a prompt that only finishes if someone already paid for a model.

You close the lid. You already know the onsite will be a tour of a demo, not a review of a system.

That scene is the hiring version of a fight the industry is having in public. People are shipping vibes and calling the result engineering. Models are also getting good at the tests we leave in the zip, which means a green bar is no longer evidence that anyone understood the constraint. If you are hiring someone who will design AI-assisted work, the take-home cannot be "build a cute agent." It has to be a spec with teeth.

Here is the contract I want in the zip. Four files, four jobs. PROMPT.md is what a stranger can finish without a paid key. RUBRIC.yml is how a second reviewer scores the same zip without a hallway chat. sample_solution/ is a path you can replay. FAILURES.md is the autopsy written before the happy path, not after the candidate notices you are watching.

If any one of those is missing, you are not grading engineering. You are grading presentation.

The prompt you actually send

Keep the story small. You are not hiring someone to invent a platform. You are hiring someone who can bound a model, refuse to log a secret, and leave a receipt a human can read on Monday.

Drop this into PROMPT.md and do not decorate it.

# Take-home: bounded quote proxy

Build a tiny HTTP service that accepts POST /v1/quote
with JSON {"prompt": "..."} and returns JSON:

{
  "ok": true,
  "text": "...",
  "route": "free",
  "bytes_out": 0,
  "truncated": false
}

Constraints (all of these are graded):

1. The service must call an LLM only through $FREE_ENDPOINT.
   If that env var is missing, exit 2 before binding the port.
2. Never write Authorization, Cookie, or api_key values to
   stdout, stderr, or the receipt file.
3. Cap model output at 500 characters. If the model returns
   more, truncate, set truncated=true, still return HTTP 200.
4. Write receipts/last.json atomically (write temp, then rename).
5. Include FAILURES.md that names at least five ways this
   design lies, including one you could not fix in the timebox.

Timebox: three hours. Do not add a UI. Do not add auth.
Do not add a second model route.

Notice what the prompt does not do. It does not ask for a framework. It does not ask for streaming. It does not ask the candidate to impress you with tools. It asks for a door that stays shut when the free path is missing.

That last bit is the point of the current market, not a product pitch. Interviewers have been burned by take-homes that only run on a personal key. The candidate looks fluent. Your replay dies in five minutes. You cannot tell skill from a weekend of paid tokens.

If you need a replay path that does not start at a checkout form, MonkeyCode's free model access and free server option are one way to point $FREE_ENDPOINT at something a second reviewer can actually hit. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The rest of this article still works if you use any other free endpoint you already trust. The method is the four files, not the vendor.

The rubric that travels with the zip

A rubric that lives in your head is a vibe. Put the scores in the zip so a skeptical teammate can grade without you in the room.

# RUBRIC.yml
version: 1
max_points: 20
timebox_hours: 3
weights:
  boot_without_endpoint: 4   # process exits 2, no port bind
  secret_hygiene: 4          # no secret material in logs or receipt
  truncate_and_flag: 3       # 500-char cap, truncated=true
  atomic_receipt: 3          # temp file + rename, last.json replays
  failures_doc: 4            # five named lies, one left unfixed
  prompt_obedience: 2        # no UI, no second route, no extra auth
fail_closed:
  - paid_or_mystery_endpoint
  - missing_FAILURES_md
  - demo_only_no_sample_solution
notes: |
  Score the sample_solution directory, not a screen recording.
  If the candidate needs a paid key to finish, score is 0 on
  boot_without_endpoint even if the demo looks smooth.

Read that file out loud once. If a sentence needs you to explain it, the sentence is still a hallway chat. Tighten it until a stranger can mark the zip in twenty minutes.

A sample solution you can replay

Label this as a worked example, not a production service. It is the smallest Python that makes the rubric angry in the right places when you break it on purpose.

#!/usr/bin/env python3
"""Worked example. Not a framework. Replay with FREE_ENDPOINT set."""
from __future__ import annotations

import json
import os
import sys
import tempfile
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path
from urllib.request import Request, urlopen

CAP = 500
RECEIPT = Path("receipts/last.json")
SECRET_KEYS = ("authorization", "cookie", "api_key", "x-api-key")


def die(code: int, msg: str) -> None:
    sys.stderr.write(msg + "\n")
    raise SystemExit(code)


def endpoint() -> str:
    url = os.environ.get("FREE_ENDPOINT", "").strip()
    if not url:
        die(2, "FREE_ENDPOINT missing; refusing to bind")
    return url


def redact(obj):
    if isinstance(obj, dict):
        out = {}
        for k, v in obj.items():
            if str(k).lower() in SECRET_KEYS:
                out[k] = "[redacted]"
            else:
                out[k] = redact(v)
        return out
    if isinstance(obj, list):
        return [redact(x) for x in obj]
    return obj


def write_receipt(payload: dict) -> None:
    RECEIPT.parent.mkdir(parents=True, exist_ok=True)
    data = json.dumps(redact(payload), indent=2).encode()
    fd, tmp = tempfile.mkstemp(dir=str(RECEIPT.parent), suffix=".tmp")
    try:
        os.write(fd, data)
        os.fsync(fd)
    finally:
        os.close(fd)
    os.replace(tmp, RECEIPT)


def call_model(prompt: str) -> str:
    body = json.dumps({"prompt": prompt}).encode()
    req = Request(endpoint(), data=body, method="POST")
    req.add_header("Content-Type", "application/json")
    with urlopen(req, timeout=30) as resp:
        raw = json.loads(resp.read().decode() or "{}")
    text = str(raw.get("text") or raw.get("content") or "")
    return text


class Handler(BaseHTTPRequestHandler):
    def log_message(self, fmt: str, *args) -> None:
        # Keep the default logger from echoing headers.
        sys.stderr.write("%s - %s\n" % (self.address_string(), fmt % args))

    def do_POST(self) -> None:
        if self.path != "/v1/quote":
            self.send_error(404)
            return
        n = int(self.headers.get("Content-Length") or 0)
        incoming = json.loads(self.rfile.read(n) or b"{}")
        prompt = str(incoming.get("prompt") or "")
        text = call_model(prompt)
        truncated = len(text) > CAP
        if truncated:
            text = text[:CAP]
        payload = {
            "ok": True,
            "text": text,
            "route": "free",
            "bytes_out": len(text.encode()),
            "truncated": truncated,
        }
        write_receipt(payload)
        blob = json.dumps(payload).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(blob)))
        self.end_headers()
        self.wfile.write(blob)


if __name__ == "__main__":
    url = endpoint()
    server = ThreadingHTTPServer(("127.0.0.1", 8088), Handler)
    sys.stderr.write("listening on 8088 via %s\n" % url.split("?", 1)[0])
    server.serve_forever()

Replay it like a skeptic, not like a fan. The commands below are the interview. If they only work on the candidate's machine, the zip is a souvenir.

unset FREE_ENDPOINT
python3 sample_solution/server.py; echo exit:$?
# expect: exit 2, nothing listening on 8088

export FREE_ENDPOINT="http://127.0.0.1:9/does-not-matter"
python3 - <<'PY'
import json, os, urllib.request
print("endpoint configured:", bool(os.environ.get("FREE_ENDPOINT")))
PY

# With a real free endpoint in FREE_ENDPOINT:
curl -sS localhost:8088/v1/quote \
  -H 'Content-Type: application/json' \
  -d '{"prompt":"Reply with 800 characters of the letter a."}'
cat receipts/last.json

The first command is the whole philosophy. Missing env, no bind, non-zero exit. A candidate who "fixes" that by silently falling back to a paid key has failed the prompt even if the JSON looks pretty.

The autopsy is the work

Ask them to write FAILURES.md before they chase green. People who start with the demo will invent failure modes that their code already avoids. That is fan fiction. You want the lies that are still in the design.

A strong autopsy reads like this, in prose, not as a trophy list.

The proxy trusts whatever JSON the free endpoint returns. If that server wraps the model text in a different key, text becomes empty and ok stays true. The receipt then looks healthy while the caller got silence. Truncation is counted in characters, not tokens and not graphemes, so a prompt that returns emoji or CJK will pass the 500 cap and still blow a downstream UI. urlopen follows redirects. A compromised endpoint can 302 the prompt at a host that is not free, and the receipt will still say "route": "free" because that string is a label, not a measurement. ThreadingHTTPServer will overlap two writes if you skip the temp-file rename under load; last.json can tear. Logging self.address_string() is harmless until someone puts a bearer token in a query string, at which point stderr becomes a secret store. None of those are clever. All of them show up in real take-homes when the model is treated like a clean function.

The line you should not forgive is the unfixed one. If FAILURES.md claims the design is complete, the candidate did not look. Engineering is the leftover risk, named in a file a stranger can read.

How this dies in review

You will see the same collapses. The zip contains a screen recording and no sample_solution/. The prompt grew a second route named premium because the candidate wanted a better answer. FAILURES.md is a paste of generic LLM risks that never mention this HTTP handler. The receipt dumps the full request headers. The server binds even when FREE_ENDPOINT is empty, then fails later in a stack trace. Tests assert only that HTTP 200 happened, which is how a model outgrows the suite: it learned the shape of your asserts and stopped touching the constraint.

Treat those as automatic zeros on the matching rubric keys. Do not debate them in the onsite. The onsite is for the one leftover risk they could not fix. Ask them to walk the redirect lie, or the character-versus-token lie, with the receipt file open. If they cannot, they shipped a vibe.

Who should not use this

Do not send this packet for a staff role whose job is system design across six services. Three hours and a quote proxy will insult them, and they will be right. Do not send it if nobody on your side will rerun the sample. A rubric you never execute is costume jewelry. Do not send it to candidates you cannot pay for the timebox. A take-home is still labor.

Skip it if your company cannot offer any free replay path. The whole point is that the interviewer and the candidate share a machine story. If the only model you believe in is a paid one, this prompt becomes theater, and you should interview some other way.

The method also fails closed for people who want a cinematic agent demo. That is the feature. You are not grading whether the model sounded clever. You are grading whether the candidate can name the ways it breaks, bound the blast radius, and leave four files a stranger can grade after midnight.

If you try the packet, change the prompt's domain so it is not this quote proxy. Keep the four names. Keep the fail-closed boot. Keep the autopsy. That is the work, and it still holds if you strip every product name out of the README.

Give Them a Prompt They Can Finish Without a Paid Key

Morgan Zhou — Thu, 17 Sep 2026 05:07:40 +0000

You unzip two take-homes on a Sunday night. Both folders look staff-level. Tests are green. The README has the calm voice of someone who has already shipped the same service twice.

Then you see the part no rubric mentions. One transcript is a long paid-model session. The other candidate stopped when a trial key died. You did not mean to hire a credit card. You hired one anyway.

That is the hole in a lot of “AI-friendly” interviews in 2026. The task looks technical. The hidden filter is billing. If the only way to finish your take-home is a personal API key, you are scoring subscription access, not judgment.

So stop sending a repo tour. Send a prompt packet: the task, the contract you will grade, a sample path, and the failure modes that still fail when the code is pretty. Make the packet runnable on a free coding-agent runtime so every candidate starts from the same side of the paywall.

The scene you are actually testing

You are not testing whether someone can vibe a CRUD app. You are testing whether they can keep an assistant inside a written contract when the clock is real and the machine is ordinary.

Picture a candidate on a laptop that is also playing music. They paste your prompt into an agent. The agent offers to “just rewrite the module.” A weak candidate says yes. A stronger one pastes the contract back into the thread and makes the agent argue with the file, not with a marketing sentence.

That difference does not show up in a polished demo. It shows up in a packet you wrote before anyone cloned the repo.

The prompt you actually send

Here is a proposed take-home, not a war story. You can drop this file in the zip as PROMPT.md and refuse to answer clarifying questions that the file already covers.

# Take-home: contract_check

You will implement a small CLI named `contract_check`.

Time box: one sitting. Do not build a platform.
Runtime: you may use an AI coding assistant. You may not use a paid API key
as a requirement of the solution. If your usual tool needs billing, switch
to a free model/runtime or write the code yourself.

## Behavior
Read a JSON contract and a unified diff on stdin. Print a JSON report on
stdout. Exit 0 only when the diff honors the contract.

Contract fields:
- max_files: int
- max_diff_lines: int
- forbidden_globs: list of fnmatch patterns (match against paths in the diff)
- required_test_globs: list of fnmatch patterns; at least one added/modified
  file must match

Report fields:
- ok: bool
- files_touched: int
- diff_lines: int
- forbidden_hits: list of paths
- missing_tests: bool
- notes: list of short strings

## Rules
1. Do not fetch the network during grading.
2. Do not read files outside the diff and the contract.
3. Commit TOOLS.md: which assistant, which runtime, and what you forbade it to do.
4. If you change the prompt, say so in NOTES.md. Silent prompt edits fail.

Notice what is missing. There is no “impress us.” There is no hidden production repo. The prompt is the product. If a candidate cannot finish it without buying tokens, the prompt is the thing you should rewrite, not the person.

If you need a shared place for people who do not already rent a coding agent, MonkeyCode’s free model access and free server option is one way to keep the take-home from becoming a wallet test. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Use it as a runtime, not as a brand name in the rubric. The rubric still has to fail pretty code.

A contract file, not a vibe

Ship the contract next to the prompt. Candidates should not have to guess what “small” means.

{
  "max_files": 4,
  "max_diff_lines": 120,
  "forbidden_globs": [
    "secrets/*",
    "**/.env",
    "**/id_rsa*"
  ],
  "required_test_globs": [
    "tests/test_*.py",
    "*_test.py"
  ]
}

You can narrate this in the interview later. Four files is a fence, not a style opinion. One hundred and twenty diff lines is a fence. Forbidden globs are a fence. Required tests are a fence. Fences are kinder than “use your judgment,” because judgment without a fence is just whoever talks first.

Sample solution (labeled, not claimed production)

This is a sample path a candidate might commit. It is intentionally boring. Boring is the point. If your assistant cannot stay boring, that is data.

#!/usr/bin/env python3
"""contract_check.py — sample take-home solution, not a library."""
from __future__ import annotations

import fnmatch
import json
import re
import sys
from pathlib import Path

FILE_RE = re.compile(r"^(?:--- |\+\+\+ )([ab]/)?(.+)$")


def parse_diff(text: str) -> tuple[list[str], int]:
    files: list[str] = []
    diff_lines = 0
    for raw in text.splitlines():
        if raw.startswith("--- ") or raw.startswith("+++ "):
            m = FILE_RE.match(raw)
            if not m:
                continue
            path = m.group(2).strip()
            if path == "/dev/null":
                continue
            if path not in files:
                files.append(path)
            continue
        if raw.startswith("+++") or raw.startswith("---"):
            continue
        if raw.startswith("+") or raw.startswith("-"):
            if raw.startswith("+++") or raw.startswith("---"):
                continue
            diff_lines += 1
    return files, diff_lines


def grade(contract: dict, diff_text: str) -> dict:
    files, diff_lines = parse_diff(diff_text)
    forbidden = []
    for path in files:
        for pat in contract["forbidden_globs"]:
            if fnmatch.fnmatch(path, pat):
                forbidden.append(path)
                break
    missing_tests = not any(
        fnmatch.fnmatch(path, pat)
        for path in files
        for pat in contract["required_test_globs"]
    )
    notes = []
    if len(files) > contract["max_files"]:
        notes.append("too many files")
    if diff_lines > contract["max_diff_lines"]:
        notes.append("diff too large")
    if forbidden:
        notes.append("forbidden path")
    if missing_tests:
        notes.append("no test file in diff")
    ok = not notes
    return {
        "ok": ok,
        "files_touched": len(files),
        "diff_lines": diff_lines,
        "forbidden_hits": forbidden,
        "missing_tests": missing_tests,
        "notes": notes,
    }


def main() -> int:
    if "--contract" not in sys.argv:
        print("usage: contract_check.py --contract contract.json < patch.diff", file=sys.stderr)
        return 2
    path = Path(sys.argv[sys.argv.index("--contract") + 1])
    contract = json.loads(path.read_text())
    report = grade(contract, sys.stdin.read())
    json.dump(report, sys.stdout, indent=2)
    sys.stdout.write("\n")
    return 0 if report["ok"] else 1


if __name__ == "__main__":
    raise SystemExit(main())

A candidate should be able to run it without a story:

python3 contract_check.py --contract contract.json < fixtures/good.patch
echo $?
python3 contract_check.py --contract contract.json < fixtures/touches_env.patch
echo $?

If those two commands do not exist in the zip, you are grading conversation again. Conversation is a demo. Exit codes are a receipt.

The rubric lives next to the prompt

Do not keep the score in a private doc the candidate never sees. That is how you accidentally grade confidence. Put a machine-readable rubric in the zip and use it as the only scoring sheet.

# rubric.yaml — proposed scoring, 10 points
contract_honored: 4   # exit 1 on forbidden path / oversized diff / missing test
no_network: 2         # grader runs with network blocked
tools_md: 2           # assistant, runtime, and refusals written down
notes_md: 1           # silent prompt edits are a zero on this row
boring_code: 1        # extra frameworks score zero, not extra credit

You can still talk in the debrief. Talk after the numbers. If a candidate used a free runtime and left TOOLS.md looking like a crime scene — “I forbade file deletes, I forbade .env, I reran the failing fixture” — that is the interview. If they used a paid model and left TOOLS.md empty, you learned something cheaper than a loop.

A tiny lock for the “no network” row keeps you honest:

# proposed local gate; not a production sandbox
python3 - <<'PY'
import socket, sys
socket.socket = lambda *a, **k: (_ for _ in ()).throw(RuntimeError("network blocked"))
sys.argv = ["contract_check.py", "--contract", "contract.json"]
import runpy
runpy.run_path("contract_check.py", run_name="__main__")
PY

If their CLI imports requests “just in case,” the gate throws. Pretty code that phones home fails the same way ugly code does. That is the whole job of a take-home in an agent year.

Failure modes you write before you meet them

Write the failures as fixtures, not as vibes in a hiring doc. A fixture is a short story the agent cannot charm.

The first failure is the secret path. The diff looks like a test fix. Buried in it is --- a/.env and a single plus line. Candidates who grade “does it compile?” will ship it. Candidates who grade the contract will not. Name the file fixtures/touches_env.patch and keep it in the zip.

The second failure is the generous rewrite. The assistant replaces four modules because “the structure was unclear.” files_touched walks past max_files while every function has a docstring. This is the 2026 version of a candidate who refactors your interview to avoid the bug. If your rubric cannot fail a beautiful diff, you do not have a rubric.

The third failure is the missing test that is not missing if you squint. They add test.md with a paragraph about coverage. required_test_globs does not match. Argue in the debrief if you want. Do not argue during scoring.

The fourth failure is the silent prompt edit. They delete the no-network rule because the agent kept apologizing. NOTES.md is empty. This is not creativity. This is changing the exam while you take it. Score that row zero even if contract_check is perfect.

The fifth failure is the paid-key dependency that sneaks back in. A wrapper that will not start without OPENAI_API_KEY is not a solution to this packet. It is a polite way of saying other people cannot run the homework. Fail it. You asked for a checker, not a vendor SDK.

What this packet is not

This is not a staff-plus architecture interview. If you need someone to unstick a multi-region outage, do not pretend a 120-line diff will tell you. It will not.

This is also not a license to offload proprietary product code onto a shared server. If your legal team does not want candidate patches on an external runtime, do not use one. Run the same prompt on a laptop you imaged yourself. The prompt still works. The fairness argument still works. The only thing that changes is who owns the disk.

Do not use this approach for roles where AI assistance is forbidden. A prompt that says “you may use an assistant” is a different exam from “you may not.” Mixing them is how you get a week of disputes.

And do not treat a free runtime as a quality oracle. Free model access can be enough to finish a fenced CLI and still be the wrong tool for a gnarly production bug. The point is equality of entry, not a ranking of vendors. If the packet only passes when the candidate buys a better model, you wrote a shopping list.

How you run the debrief

Open TOOLS.md first. Ask what they forbade the agent to do. Then open the failing fixture they did not expect. Ask whether they would loosen max_diff_lines or keep it. You are listening for whether they can defend a fence.

Keep the paid-transcript flex out of the room. If two people finished on a free server and one person finished on a personal key, score the contract, not the invoice. The industry is already loud about whether AI codes “better than most developers.” Your take-home does not need to join that argument. It needs to tell you if this person can keep a tool inside a file they agreed to.

If you want every candidate on the same side of the paywall, run the packet on a free model and a free server once yourself before you send it. When the prompt still holds on that runtime, you are hiring a person. When it only holds on a billed session, you are hiring a plan.

Put the Rubric in the Take-Home Zip

Morgan Zhou — Wed, 16 Sep 2026 03:32:05 +0000

You open the candidate zip at 10:02. There is no LRU cache. There is no URL shortener. There is a prompt, a rubric, a sample solution that is allowed to look like a first draft, and a short file named failure modes. That is the interview.

A coding agent can paint a pull request green. You have watched it happen on a quiet laptop with a paid seat. The hire you actually need is the person who can say what green was supposed to mean when the model is whoever is free today and the box is not theirs.

If your take-home still asks for a working function, you are grading autocomplete. Ask for the four files instead. Then run them on one shared endpoint so you are not grading wallets.

The packet, not the demo

Picture a hallway conversation after an onsite. Someone says the agent nailed the refactor. Someone else says it burned the shared budget and labeled every timeout as INFO. Both can be true. The missing object is the contract: what the agent was allowed to send, what it was forbidden to invent, and how you would know it cheated the meter.

You are not hiring a magician. You are hiring the person who can write that contract before the model starts typing.

The four files are the whole take-home. A prompt the agent will actually see. A rubric a human can score in fifteen minutes. A sample solution that is allowed to be ugly. A failure catalog that names the ways this task dies in the wild. If a candidate cannot produce those four, a green PR is a costume.

This is a worked example, not a war story from a specific hiring loop. Run it before you ship it. The inner bug is small on purpose. The outer task is whether they can specify the work.

Pin the box before you pin the grade

You cannot compare two packets if one candidate ran a frontier model on a personal GPU and the other pasted into a free chatbot on a phone. That is not an interview. That is a hardware raffle.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option are useful here for one boring reason: every candidate hits the same remote endpoint and the same machine. You still own the prompt, the tests, and the rubric. The product is the pin, not the punchline.

Treat the shared box like CI. Nobody gets to swap the runner because their laptop is nicer. Point LLM_ENDPOINT at the free server, keep API keys out of the zip, and grade the files they wrote.

File one: the prompt the agent will see

Here is a candidate-facing prompt you can drop in PROMPT.md. It is intentionally bounded. An unbounded prompt is how a take-home turns into a weekend.

You are fixing classify_logs.py on the shared server.

Goal: label each log line ERROR, WARN, or INFO by calling
$LLM_ENDPOINT (OpenAI-style /v1/chat/completions).

Hard limits:
- Do not send the whole file on every line.
- Identical lines must not pay for a second model call.
- If the endpoint is down, fail the run. Do not default to INFO.
- No extra network. No extra files outside /work.
- Stop after 8 model calls that do not change a test.

Deliver: a patch, pytest passing, and a 12-line note on what you refused to send.

Read it out loud. If you cannot hear the stop condition, the agent will keep looping until the free server sighs. The prompt is part of the grade. A novel is a fail.

File two: the rubric you will actually use

Keep the rubric short enough to print. Long rubrics become fan fiction. This one fits on a card.

PROMPT SHAPE (0-3)
  3 bounded goal, hard limits, stop rule
  1 vibes and "just make it work"

TOKEN SHAPE (0-3)
  3 per-call payload is a line (or a tiny window), not the file
  1 whole log hitchhikes on every request

FAILURE (0-2)
  2 endpoint down => non-zero exit, no fake INFO
  0 swallows the error

SAMPLE (0-2)
  2 tests would catch the sample if you broke it
  0 sample is a screenshot of a chat

FAILURE CATALOG (0-2)
  2 names two real death modes with a signal you can grep
  0 "it might be slow"

Hire bar: 9/12. A 3 on token shape with a 0 on failure is a no.

Notice what is missing. There is no extra credit for a named model. There is no bonus for a cinematic README. You are scoring whether they can keep a free shared endpoint alive.

File three: the sample, and the bug it has to kill

The inner program is a log classifier that looks harmless. It is not. It sends the entire file with every line, so fifty lines means fifty copies of the same novel. On a free model that is how you turn a take-home into a brownout.

# classify_logs.py — broken on purpose. Example only; run it yourself.
from __future__ import annotations

import json
import os
import sys
import urllib.error
import urllib.request

ENDPOINT = os.environ.get("LLM_ENDPOINT", "http://127.0.0.1:8080/v1/chat/completions")


def classify_line(line: str, whole_file: str) -> str:
    payload = {
        "messages": [
            {"role": "system", "content": "Label the line ERROR, WARN, or INFO. One word."},
            {"role": "user", "content": whole_file + "\n\nNow label this line:\n" + line},
        ]
    }
    req = urllib.request.Request(
        ENDPOINT,
        data=json.dumps(payload).encode("utf-8"),
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=20) as resp:
        body = json.loads(resp.read().decode("utf-8"))
    return body["choices"][0]["message"]["content"].strip()


def classify_file(path: str) -> list[str]:
    text = open(path, encoding="utf-8").read()
    return [classify_line(line, text) for line in text.splitlines() if line]


if __name__ == "__main__":
    print("\n".join(classify_file(sys.argv[1])))

A candidate who writes a sample that still concatenates whole_file has not seen the problem. They have seen a chat window. Your sample has to be worse than that, and then slightly better.

A fair sample solution does three dull things. It sends the line, maybe a three-line window. It caches identical lines. It refuses to invent INFO when the socket dies. Regex short-circuit is allowed. Cleverness is not required.

# classify_logs_sample.py — example fix, not a library.
from __future__ import annotations

import json
import os
import re
import urllib.error
import urllib.request

ENDPOINT = os.environ.get("LLM_ENDPOINT", "http://127.0.0.1:8080/v1/chat/completions")
LOCAL = re.compile(r"\b(ERROR|WARN|INFO)\b")


def classify_line(line: str, cache: dict[str, str]) -> str:
    if line in cache:
        return cache[line]
    m = LOCAL.search(line)
    if m:
        cache[line] = m.group(1)
        return cache[line]
    payload = {
        "messages": [
            {"role": "system", "content": "Reply ERROR, WARN, or INFO. One word."},
            {"role": "user", "content": line[:500]},
        ]
    }
    req = urllib.request.Request(
        ENDPOINT,
        data=json.dumps(payload).encode("utf-8"),
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    try:
        with urllib.request.urlopen(req, timeout=20) as resp:
            body = json.loads(resp.read().decode("utf-8"))
    except urllib.error.URLError as exc:
        raise SystemExit(f"endpoint down: {exc}") from exc
    label = body["choices"][0]["message"]["content"].strip().split()[0]
    if label not in {"ERROR", "WARN", "INFO"}:
        raise SystemExit(f"bad label: {label!r}")
    cache[line] = label
    return label

The sample is a teaching tool. If their sample is longer than the prompt, they are hiding. If their sample still needs a named paid model to make sense, they failed the environment.

The test that makes the rubric real

A rubric without a meter is a speech. This test does not care what the model said. It cares how fat each request was. Fifty lines should not ship fifty copies of the novel.

# test_token_shape.py — run with: python -m pytest test_token_shape.py -q
import json
import classify_logs as cl

class FakeHTTP:
    def __init__(self):
        self.bytes_out = 0
        self.calls = 0

    def __call__(self, req, timeout=20):
        self.calls += 1
        self.bytes_out += len(req.data or b"")
        class Resp:
            def read(self_inner):
                return json.dumps(
                    {"choices": [{"message": {"content": "INFO"}}]}
                ).encode()
            def __enter__(self_inner): return self_inner
            def __exit__(self_inner, *a): pass
        return Resp()

def test_payload_stays_near_the_line(monkeypatch, tmp_path):
    log = tmp_path / "app.log"
    log.write_text("\n".join(f"INFO boot {i}" for i in range(50)) + "\n")
    fake = FakeHTTP()
    monkeypatch.setattr(cl.urllib.request, "urlopen", fake)
    cl.classify_file(str(log))
    avg = fake.bytes_out / max(fake.calls, 1)
    assert avg < 800, f"average payload {avg:.0f}b; the whole file is hitching a ride"

def test_down_endpoint_is_not_info(monkeypatch, tmp_path):
    def boom(req, timeout=20):
        raise cl.urllib.error.URLError("refused")
    monkeypatch.setattr(cl.urllib.request, "urlopen", boom)
    log = tmp_path / "app.log"
    log.write_text("kernel panic\n")
    try:
        cl.classify_file(str(log))
    except SystemExit:
        return
    raise AssertionError("down endpoint must not become a fake INFO")

Run it locally first, then on the shared server with the free model endpoint filled in. If pytest is green on a laptop and red on the pin, believe the pin. That gap is the interview.

python -m pytest test_token_shape.py -q
LLM_ENDPOINT=http://127.0.0.1:8080/v1/chat/completions \
  python classify_logs.py /tmp/app.log; echo exit:$?

You are not measuring tokens with a vendor dashboard. You are measuring bytes on the wire. Bytes do not care about the marketing name of the model.

File four: how this task actually dies

Failure modes are not a vibe. They are scenes with a grep.

One death is the novel-in-the-payload. The signal is average request size climbing with file length. If doubling the log doubles the payload, the agent did not read the prompt. It glued.

Another death is courtesy INFO. The endpoint is down, the function returns INFO, and a pager stays quiet. The signal is a zero exit after a refused connection. You already have a test for that. If their catalog does not mention it, they wrote a happy path.

A third death is cache blindness. The log is 4,000 copies of the same line. The agent still makes 4,000 calls. The free server is not angry at the candidate. It is gone.

A fourth death is the prompt with no stop. The agent retries, reformats, retries, writes a new helper, retries. Eight useless calls was the rail. Without it you are grading stamina.

Candidates fail this take-home in patterned ways. They ship a rubric that only says tests pass. They paste a two-thousand-word prompt that never names the endpoint. Their sample still sends whole_file. They assume a specific paid model and leave LLM_ENDPOINT unset. They never run on the shared box, so they never see the timeout that their laptop hid.

When you score, read the failure catalog before the patch. People reveal what they fear. If they only fear slow CI, they have not met a shared free model yet.

What you do in the debrief

Do not watch them drive the agent live. That is theatre. Sit with the four files. Ask why the prompt forbids extra network. Ask which test would still pass if the sample started concatenating the file again. If they cannot point at avg < 800, they cannot defend the rubric.

Then ask what they would cut if the free server got noisier tomorrow. The right answer is payload and retries, not a shopping list of bigger models. Constraint is the point. Anyone can look smart with an unlimited meter.

Limitations, and who should not use this

This packet is for roles that will specify agent work: platform, SRE-adjacent, eval, internal tools. It is a poor filter for a junior who has never seen an HTTP client. It is a worse filter for a staff engineer whose job is systems design and who will not touch a model that week.

Do not use it as a secret IQ test. The inner bug is obvious once you look at whole_file. The outer skill is writing the rails. If you hide the rails, you are grading luck.

A free shared server is noisy. Neighbors exist. Timeouts exist. That is a feature for this interview and a problem for a production classifier. Do not promote the sample to a service. Do not claim a latency number you did not measure. Do not bake a model name into the rubric; names move and the pin should not.

If your company already standardizes on one paid seat for every candidate, you may not need a free endpoint. You still need the four files. The zip is the method. The free model and free server are only how you stop the raffle.

Ship the zip with a dead classifier, a live test, and a rubric that fits on a card. If you want every candidate on the same free model and the same free server while they write it, MonkeyCode can be that pin. Keep the rest in git. Hire the person whose failure modes you could grep.

Grade the Receipt, Not the Agent Demo

Morgan Zhou — Tue, 15 Sep 2026 03:09:21 +0000

Grade the Receipt, Not the Agent Demo

It is 11:52 PM and the submission is a nine-minute screen recording. You watch an agent read an issue, patch two files, run the test suite, and go green. You watch it twice. Then you close the tab, because you still do not know what the run cost, how many model calls it took, or whether any path other than the happy one was ever exercised.

The demo answers a narrow question: can this thing run once, on the author's laptop, with the author watching? That is not the question a take-home is supposed to answer. A take-home is a measurement instrument, and an instrument that only works while its owner holds it is not measuring anything.

Make the deliverable a receipt

When agent-shaped take-homes started showing up everywhere, the quality of the demos went up and the quality of the evidence went down. Candidates learned to make a loop that looks autonomous; reviewers learned to nod at a terminal recording. Nothing in that exchange proves the candidate can build something that stays inside a constraint.

So change the deliverable. Stop asking for a working demo and ask for a frozen ledger — a file the reviewer can recompute in under a minute and compare against what the candidate claimed. Everything else in the task hangs off that.

Here is the whole spec, short enough to paste into a message:

Build a tool that resolves fixtures/issue_142.json.
Hard budget: 40,000 tokens, 25 tool calls, 300 seconds wall clock.
Emit one JSON object per model call to usage.jsonl,
with prompt_tokens and completion_tokens.
If you hit a cap, stop and write give_up.json. A clean stop counts as a pass.
Deliver: agent.py, the usage.jsonl from your own run, and a ten-line note
naming one thing your agent cannot do.

Three numbers, one file format, one admission of weakness. That is the entire assignment.

The harness that makes the receipt checkable

You cannot grade a ledger you cannot regenerate. Write a small runner that executes the candidate's command under the same caps you published, then freezes the result. This is the version I hand to reviewers:

#!/usr/bin/env python3
"""budget_run.py — execute a submission under a fixed budget and freeze the receipt."""
import hashlib, json, pathlib, subprocess, sys, time

SPEC = {"token_cap": 40_000, "wall_seconds": 300, "tool_call_cap": 25}

def read_usage(path):
    if not pathlib.Path(path).exists():
        return 0, 0
    calls = [json.loads(l) for l in pathlib.Path(path).read_text().splitlines() if l.strip()]
    tokens = sum(c.get("prompt_tokens", 0) + c.get("completion_tokens", 0) for c in calls)
    return tokens, len(calls)

def digest():
    h = hashlib.sha256()
    for p in sorted(pathlib.Path(".").glob("*.py")):
        h.update(p.read_bytes())
    return h.hexdigest()[:16]

def main():
    started = time.monotonic()
    try:
        proc = subprocess.run(sys.argv[1:], capture_output=True, text=True,
                              timeout=SPEC["wall_seconds"])
        timed_out, code = False, proc.returncode
    except subprocess.TimeoutExpired:
        timed_out, code = True, None
    wall = round(time.monotonic() - started, 2)
    tokens, calls = read_usage("usage.jsonl")
    flags = [name for name, bad in {
        "tokens": tokens > SPEC["token_cap"],
        "wall_seconds": wall > SPEC["wall_seconds"],
        "tool_calls": calls > SPEC["tool_call_cap"],
        "missing_usage_file": calls == 0,
        "timeout": timed_out,
    }.items() if bad]
    ledger = {"cmd": sys.argv[1:], "wall_seconds": wall, "tokens": tokens,
              "tool_calls": calls, "exit_code": code, "over_budget": flags,
              "digest": digest()}
    pathlib.Path("ledger.json").write_text(json.dumps(ledger, indent=2))
    print(json.dumps(ledger, indent=2))
    return 1 if flags else 0

if __name__ == "__main__":
    sys.exit(main())

A reviewer runs one line, twice: once against the submitted artifact, once against a fixture the candidate has never seen. python budget_run.py python agent.py --task fixtures/issue_142.json. If the digest changes between runs, the candidate modified code between the claim and the check, and the conversation starts there instead of in the comments.

A sample solution that stops on purpose

The interesting part of a good submission is not the loop; it is the guard in front of the loop. This is roughly what a passing solution looks like, and the failure branch matters more than the success branch:

# agent.py — spend first, then act. Every model call leaves a receipt.
import json, pathlib, sys

CAP = 40_000
USAGE = pathlib.Path("usage.jsonl")

def spent():
    if not USAGE.exists():
        return 0
    return sum(json.loads(l)["prompt_tokens"] + json.loads(l)["completion_tokens"]
               for l in USAGE.read_text().splitlines() if l.strip())

def call_model(client, messages, tools):
    if spent() + estimate(messages) > CAP:          # estimate() is your tokenizer
        pathlib.Path("give_up.json").write_text(
            json.dumps({"reason": "token_cap", "spent": spent()}))
        sys.exit(0)                                  # clean stop, not a crash
    reply = client.chat(messages=messages, tools=tools)
    with USAGE.open("a") as f:
        f.write(json.dumps({"prompt_tokens": reply.usage.prompt_tokens,
                            "completion_tokens": reply.usage.completion_tokens}) + "\n")
    return reply

Notice that exhaustion is a first-class outcome. A candidate who wrote this has understood that budget is a design constraint; a candidate who retries fourteen times after a throttling response has not, and the ledger shows it without you having to argue about it.

The rubric, weighted for evidence

Keep it to five rows. Longer rubrics collapse into style grading.

Criterion	Weight	A pass looks like
Reproducibility	25	Your one-line invocation exits 0 on a clean checkout
Budget honesty	25	Your `ledger.json` and their claimed spend agree within 5%
Failure behavior	20	The over-budget run stops and writes `give_up.json` instead of thrashing
Held-out task	20	The agent clears a fixture it has never seen
Write-up	10	The ten-line note names a real limitation of its own approach

Everything above is checkable. Nothing depends on whether you liked the candidate's variable names.

The failure modes you will actually see

The first is the missing receipt: an agent that works, plus no usage.jsonl. That is not a formatting slip — it means the candidate never treated spend as part of the deliverable. The second is the retry storm, where a throttled call triggers a loop that quietly eats the entire cap. The third is fixture hardcoding, which is exactly why the held-out task carries twenty points.

The fourth is the interesting one: success by deletion. The agent "fixes" the failing test by removing the assertion. Grade the diff, not the exit code, and this one surfaces in seconds.

The fifth is machine drift, and it is the one worth designing out before you grade anyone. Give candidates a fixed environment instead of asking them to describe their laptop, because a submission that only runs on the author's box is not a submission. The open-source MonkeyCode project is one option here, and it is the one I point candidates at, since it offers a free tier and a hosted server so nobody's hardware decides their score. The project's outreach states a free allowance of ten million tokens plus a free server option; terms like that move, so read the current ones before you write a number into a rubric. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Limits, and who should skip this

This harness measures tokens, wall clock, and tool calls. It does not measure correctness — held-out tests still do that, and without them you are grading thrift. It also cannot prove the candidate didn't call some unlogged endpoint from a subprocess; it only makes hiding expensive and visible.

If your organization forbids third-party network access during interviews, do not use a hosted tier, free or otherwise. If you cannot freeze a fixture set, skip the held-out row and be honest that you are grading reproducibility only. And if your team has no fixed budget to publish, the whole exercise is theater — a take-home without a constraint is a portfolio review with extra steps.

If you want to dry-run the harness before you spend your own credits on it, start on MonkeyCode's free tier and check whether your ledger.json matches what the dashboard reports. A five-minute discrepancy hunt will teach you more about your rubric than another round of demo videos.

MonkeyCode provides free models that can run this workflow.