The Take-Home That Runs Without Your Keys

You open the folder on a Monday morning. Twenty take-home submissions, each with a README, and three that refuse to start. One wants an API key you do not have. Another assumed python3.11 while your box ships 3.12. The third wrote its trace to /Users/dana/tmp/trace.jsonl, a path that has never existed on anyone else's laptop. By eleven o'clock you are not judging engineering. You are triaging ModuleNotFoundError, which is a different job with worse pay.

The reflex is to add a line to the README: please make sure it runs. That fixes nothing. You are asking the candidate to simulate your machine, and they cannot see it. Change the task instead, so that the default path runs with nothing but a checkout, and change the rubric so you score the run rather than the prose around it.

Make the keyless path the main path

Hand this prompt over verbatim. It is small on purpose: four requirements, one timebox, no room left to argue about what "done" means.

Build a CLI, triage.py, that walks a directory and labels each file as
keep, archive, or review.

1. `python triage.py --root <dir> --dry-run --trace <file>` must run with no
   network access and no credentials, and must still write one JSON line per
   file.
2. With BASE_URL, API_KEY, and MODEL present in the environment, the tool may
   call that chat completions endpoint to improve its labels.
3. Nothing outside --root (read) and --trace (write) may be touched.
4. --limit N caps outbound model calls, and the trace must report how many
   were used, per row.

Timebox: three hours. Partial credit is real. Tell us what you skipped.

The dry-run clause is load-bearing. A reviewer can grade structure, boundaries, and trace shape before spending anything at all, and a candidate without a funded provider account can still pass. That second effect is the interesting one: if your exercise quietly requires a credit card, you have filtered for people who own one. The operator of MonkeyCode told me the project offers free model access and a free server option; treat those as operator-supplied claims, confirm the current terms yourself, and design the exercise so it survives either way.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The rubric is five rows, not a vibe

Write the weights down before you read a single submission. Otherwise the last one you read wins, and the last one you read is usually the one submitted at 2 a.m. by someone who happens to share your taste in file names.

Weight	Row	What earns the point	The 60-second check
30%	Runs cold	starts with no env vars, no network, no side files	container run with --network none and an empty environment
25%	Boundary discipline	writes only to --trace, walks only --root	diff the mount, or read every open(..., "w")
20%	Trace fidelity	one JSON object per file, stable keys, calls_used present	wc -l against `find
15%	Failure honesty	a dead network degrades to a labelled row, not a crash	run once with {% raw %}BASE_URL pointing at a black hole
10%	Readability	you can explain the flow to a colleague after one pass	the one-pass test

Weight "runs cold" highest. It is the cheapest row to check, and it decides whether the other four rows are gradeable at all.

A sample solution, short on purpose

You do not need a framework for this. Standard library, one file, roughly eighty lines, and a fallback heuristic that keeps working when the network does not.

#!/usr/bin/env python3
"""triage.py - reference solution for the file-triage take-home."""
from __future__ import annotations

import argparse
import json
import os
import pathlib
import sys
import urllib.request

LABELS = ("keep", "archive", "review")


def walk(root: pathlib.Path):
    for path in sorted(root.rglob("*")):
        if path.is_file():
            yield path


def heuristic(path: pathlib.Path) -> tuple[str, str]:
    name = path.name.lower()
    if name.endswith((".png", ".jpg", ".zip", ".log")):
        return "archive", "binary or log-like extension"
    if name.endswith((".md", ".py", ".ts")):
        return "keep", "source or prose"
    return "review", "unrecognised extension"


def ask_model(text: str) -> tuple[str, str]:
    base = os.environ["BASE_URL"].rstrip("/")
    payload = json.dumps({
        "model": os.environ.get("MODEL", "default"),
        "messages": [{"role": "user", "content":
                      "Label this file keep, archive, or review.\n\n" + text[:2000]}],
    }).encode("utf-8")
    req = urllib.request.Request(
        base + "/chat/completions",
        data=payload,
        headers={"Content-Type": "application/json",
                 "Authorization": "Bearer " + os.environ["API_KEY"]},
    )
    with urllib.request.urlopen(req, timeout=30) as resp:
        raw = json.load(resp)
    answer = raw["choices"][0]["message"]["content"].lower()
    for label in LABELS:
        if label in answer:
            return label, "model"
    return "review", "unparseable model answer"


def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--root", required=True, type=pathlib.Path)
    ap.add_argument("--trace", required=True, type=pathlib.Path)
    ap.add_argument("--dry-run", action="store_true")
    ap.add_argument("--limit", type=int, default=20)
    args = ap.parse_args()

    if args.dry_run:
        allowed = 0
    elif os.environ.get("BASE_URL") and os.environ.get("API_KEY"):
        allowed = args.limit
    else:
        print("no BASE_URL/API_KEY; rerun with --dry-run", file=sys.stderr)
        return 2

    used = 0
    with args.trace.open("w", encoding="utf-8") as out:
        for path in walk(args.root):
            rel = path.relative_to(args.root)
            if used < allowed:
                try:
                    label, why = ask_model(path.read_text(errors="replace"))
                    used += 1
                except Exception as exc:  # network, auth, bad JSON, timeout
                    label, why = heuristic(path)
                    why = "model failed (%s), fell back" % type(exc).__name__
            else:
                label, why = heuristic(path)
            out.write(json.dumps({"path": str(rel), "label": label,
                                  "reason": why, "calls_used": used}) + "\n")
    print("wrote %s using %d model call(s)" % (args.trace, used))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Two details matter more than the classifier. --dry-run sets allowed = 0 and keeps walking the same loop, so the keyless path cannot silently drift away from the real one. And the bare except turns a dead network into a labelled row instead of a stack trace, which is exactly what the failure-honesty row is measuring. A candidate who writes both has been paged before, and it shows.

Grade the run, not the report

Your grader should be three commands and no opinions. It should also be the same command you hand to the candidate, so nobody has to guess what "runs" means.

TRACE_DIR=$(mktemp -d)
docker run --rm --network none -w /work \
  -v "$PWD/submission:/work:ro" \
  -v "$TRACE_DIR:/trace:rw" \
  python:3.12-slim \
  python triage.py --root /work/samples --dry-run --trace /trace/t.jsonl

python - "$TRACE_DIR/t.jsonl" <<'PY'
import json, pathlib, sys
rows = [json.loads(l) for l in pathlib.Path(sys.argv[1]).read_text().splitlines() if l.strip()]
labels = {r["label"] for r in rows}
assert labels <= {"keep", "archive", "review"}, labels
assert all(r["calls_used"] == 0 for r in rows), "dry-run must not call out"
print("ok: %d rows, labels=%s" % (len(rows), sorted(labels)))
PY

--network none is the whole trick. It converts "it works on my laptop" from a debate into an exit code, and it never needs your key or the candidate's.

The ways it breaks

Imported keys. The candidate reads os.environ["API_KEY"] at module import, so --dry-run dies before argparse ever runs. This is the most common failure by a wide margin, and it is a design smell in production code too.

Silent truncation. The tool classifies the first five hundred bytes and never says so. No flag, no trace field, no comment. You find out when two unrelated large files get identical labels for no visible reason.

Trace on success only. The trace is written after the loop finishes, so an interrupt at file ninety of a hundred leaves you with nothing to grade. Streaming one row per file is one line of code and infinitely more debuggable.

Label drift. The model returns delete, or DELETE, or a sentence containing both "keep" and "archive". Without a whitelist and a fallback, the trace becomes free text wearing a JSON costume.

Bonus-point prose. The README promises rate limiting, caching, and a plugin system that no code implements. Score the repository, not the roadmap.

Who should not use this

If the role is really about writing design documents or negotiating scope, this exercise measures the wrong muscle. If you cannot run containers on your own machine, the grader above is not for you either. And a dry-run harness cannot see how someone behaves when the requirements are genuinely ambiguous, because a trace can tell you what the code did and never why it chose to. Keep a short follow-up conversation for that part; it is still the part that decides a hire.

The free server option is convenient for exactly one thing, running the grader somewhere that is not your laptop, and the free model tier is a keyless path for candidates rather than a promise your hiring process should depend on. Verify both against today's terms, keep the local command as a fallback, then spin the harness up once yourself before you ask twenty people to run it — and watch what breaks.