Send the Scorecard With the Take-Home

You open the packet at 10:40 p.m. The candidate built "an agent." The README has a GIF, a free-server URL, and three adjectives. You click the URL. It times out. The GIF cannot be graded.

That is the failure this take-home is built to prevent. You are not collecting vibes. You are collecting a scorecard that another engineer can replay on Monday without asking the candidate to narrate the demo again.

A take-home for agent work has four pieces, and they travel together. The prompt states the constraint. The rubric states what a pass looks like. A sample harness shows the shape of a valid submission. The failure notes tell the candidate which impressive-looking artifacts you will reject. Leave any one of those out and you will spend the review call reconstructing intent from a chat log.

Here is the prompt you can paste into the packet. It is deliberately narrow. A wide prompt produces a wide excuse.

Take-home prompt (48 hours, async)

Build a small grading harness for a tool-using assistant.
The assistant receives a task, may call one tool, and returns a short answer.
You do not need a clever model. You need a replayable score.

Requirements:
1. Commit fixtures: at least six tasks, each with an expected tool name and an expected answer fragment.
2. Run those fixtures against an endpoint you control.
3. Write scorecard.json. Each row records task_id, expected_tool, observed_tool, answer_ok, and provider_meta taken from the response. If the response has no model id, write null. Do not fill the gap.
4. Include a check command that fails the build when a row claims a quota, a hardware spec, or a latency SLO that is not present in provider_meta.
5. Document how a reviewer re-runs the check. Do not document a benchmark you did not run.

You may use any free model access and, if you have one, a free server so the reviewer can reach the same endpoint. Availability is not a score. A missing field is an honest null, not a reason to invent a name.

That last sentence is the whole point of the exercise. Free access is a convenience for a short review window. It is not evidence. Candidates love to write that they used "the fast model on the free tier" and then attach a score you cannot regenerate. Your packet should make that sentence uncommitable.

If they need a place to try the loop without provisioning a box, MonkeyCode's free model access and free server option are enough to stand an endpoint up for this packet. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Treat that pair of options as availability, not as a catalog. This walkthrough does not name models, quotas, hardware, or a retention window, because those were not supplied as stable facts. If the console shows a model id, the harness copies that id into provider_meta. If it does not, the field stays null.

The rubric should live in code, not in a slide. A reviewer who cannot execute the rubric will renegotiate it on the call, and then two candidates were not judged on the same task. Think of the rubric as a lock on the door. The key is the scorecard. A charming README does not pick the lock.

# UNEXECUTED EXAMPLE. Proposal only. Not a measured run.
from dataclasses import dataclass

@dataclass(frozen=True)
class Rubric:
    min_tasks: int = 6
    require_tool_match: bool = True
    require_answer_fragment: bool = True
    forbid_unsourced_claims: tuple = ("quota", "tokens_granted", "gpu", "p95_ms", "sla")

def row_passes(row: dict, rubric: Rubric) -> tuple[bool, str]:
    if rubric.require_tool_match and row.get("observed_tool") != row.get("expected_tool"):
        return False, "tool_mismatch"
    if rubric.require_answer_fragment and not row.get("answer_ok"):
        return False, "answer_fragment_missing"
    meta = row.get("provider_meta") or {}
    for key in rubric.forbid_unsourced_claims:
        if key in row and key not in meta:
            return False, f"unsourced:{key}"
    return True, "ok"

Read that slowly. The pass condition is boring on purpose. Tool match, answer fragment, and a ban on numbers the provider did not return. You are hiring for judgment under incomplete information. A candidate who adds a leaderboard column without a source has failed, even if the demo looked smooth.

The sample solution is a harness, not a model wrapper with aspirations. Label it as unexecuted when you send it. You are showing shape. You are not claiming a score. The unfinished function is a feature. It stops a reviewer from pasting this sketch into a score spreadsheet and calling it evidence.

# UNEXECUTED EXAMPLE. Sample solution shape for the take-home.
# Replace grade_one() with a real HTTP call. Do not invent provider fields.
import argparse, json, pathlib, sys

def grade_one(task: dict, endpoint: str) -> dict:
    # Proposal: POST {task_id, prompt} and read the JSON body.
    # Copy only keys the body actually contains: tool, answer, provider_meta.
    raise NotImplementedError("wire this to your endpoint")

def annotate(task: dict, observed: dict) -> dict:
    meta = observed.get("provider_meta") or {}
    answer = observed.get("answer") or ""
    row = {
        "task_id": task["id"],
        "expected_tool": task["expected_tool"],
        "observed_tool": observed.get("tool"),
        "answer_ok": task["answer_fragment"] in answer,
        "provider_meta": meta,
    }
    if row["observed_tool"] != row["expected_tool"]:
        row["reason"] = "tool_mismatch"
    elif not row["answer_ok"]:
        row["reason"] = "answer_fragment_missing"
    else:
        row["reason"] = "ok"
    return row

def check(rows: list[dict]) -> int:
    bad = [r for r in rows if r.get("reason") != "ok"]
    pathlib.Path("scorecard.json").write_text(json.dumps(rows, indent=2))
    print(f"rows={len(rows)} failed={len(bad)}")
    return 1 if bad else 0

def main() -> int:
    p = argparse.ArgumentParser()
    p.add_argument("--fixtures", required=True)
    p.add_argument("--endpoint", default="http://127.0.0.1:8787/grade")
    p.add_argument("--check-only", action="store_true")
    args = p.parse_args()
    if args.check_only:
        rows = json.loads(pathlib.Path("scorecard.json").read_text())
        return check(rows)
    tasks = [
        json.loads(line)
        for line in pathlib.Path(args.fixtures).read_text().splitlines()
        if line.strip()
    ]
    rows = []
    for task in tasks:
        try:
            observed = grade_one(task, args.endpoint)
        except NotImplementedError:
            observed = {"tool": None, "answer": "", "provider_meta": {}}
        rows.append(annotate(task, observed))
    return check(rows)

if __name__ == "__main__":
    sys.exit(main())

Pair that harness with fixtures that look like work, not like riddles. Six lines is enough to start. Each line is one contract: an id, a prompt, the tool you expect, and a fragment that must appear in the answer. If you need a seventh case, add a task whose correct behavior is to call no tool and say so. Do not hide that case. Tell the candidate it exists.

{"id":"t1","prompt":"What is the status of invoice 1044?","expected_tool":"lookup_invoice","answer_fragment":"1044"}
{"id":"t2","prompt":"Do not call tools. Reply with the word ready.","expected_tool":null,"answer_fragment":"ready"}

The commands you put in the packet should be copy-pasteable and secret-free. A reviewer should not need a key you forgot to rotate. If the endpoint wants a token, the candidate documents the env var name and you inject it locally. The repo never contains the value.

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python harness.py --fixtures fixtures/tasks.jsonl --endpoint "$EVAL_ENDPOINT"
python harness.py --check-only

If EVAL_ENDPOINT points at a free server, say so in one line of the README and stop. Do not add a hardware guess beside it. The check command is the review. The prose is optional. A green --check-only on the committed scorecard.json means the file is internally consistent. It does not mean the live URL still answers. Run the live path only when you still have the endpoint, and write a second file if the new rows differ. Never overwrite history to make Friday look like Thursday.

Walk the candidate through the failures you will actually reject, in the same voice you use on the call. A GIF with no scorecard is a pitch, not a submission. A scorecard that names a model the response never returned is a fabricated trace. A writeup that quotes a token grant, a GPU class, or a p95 you did not measure is the same failure as a broken test, because the rubric treats unsourced claims as broken rows. A harness that only runs against a laptop process and then calls that "production on the free server" has mixed two environments. Split them. Local is local. The free server is whatever answered the HTTP call, and only that response may fill provider_meta.

There is a softer failure that still wastes your Friday. The candidate returns a perfect score on six toy tasks and a paragraph about how the agent "understood intent." You asked for tool match and an answer fragment. Understanding is not in the rubric. Do not promote it during the review or you have graded a different assignment than the one you sent. That is how two finalists become incomparable. Another common miss is the silent retry. The candidate's script calls the endpoint three times and keeps the flattering row. Your packet should say one attempt per task unless provider_meta records the retry count that the server itself returned. If the server does not return that count, the harness does not invent one.

Clock skew shows up too. A candidate stamps every row with the time on their laptop and then argues the free server was "fast." Latency belongs in the row only when the response, or a timer wrapped around that single call, produced it. A hand-typed 42ms is a story. Stories go in the cover letter. They do not go in scorecard.json.

Limitations belong in the packet, not in a footnote you remember later. This approach does not measure safety, refusal quality, or cost. Six fixtures will not catch a tool that deletes the wrong row in production. A free server can disappear between Thursday and the review, so the scorecard file is the artifact of record and the live URL is a convenience. If your process requires a signed vendor, a data-processing agreement, or a fixed quota, do not use a free option as the system under test. Run the same harness against the endpoint your security review already allowed, and keep the null-filling rule. Free availability can also change without notice. Do not write a hiring policy that assumes this week's console will look the same next quarter.

Who should skip this take-home: interviewers who need a live whiteboard and a shared editor, teams hiring for research taste rather than operational grading, and anyone who will override the rubric when a charismatic README disagrees with it. Also skip it if you cannot spend thirty minutes re-running --check-only. A scorecard you will not execute is just a longer GIF. Skip it too if the role's real risk is data leaving a boundary you have not described. This packet does not teach that boundary. It only teaches you to grade what was written down.

The sample above is a proposal. It has not been executed against a live endpoint in this article, and grade_one is intentionally unfinished so you do not mistake a sketch for a result. Wire it, run the six fixtures, and commit the scorecard the command actually wrote.

If a free model path and a free server are already on the table, point EVAL_ENDPOINT there, copy only the fields the response returns, and leave the rest null. That is the submission. The rest of the pitch can wait for the call.