Give the Take-Home a Scoreboard

It is Thursday night and there are three take-homes in your review queue. Each one is a repository, a README, and a paragraph that begins "I'm confident this is production ready." You have twenty minutes per candidate. Within five minutes you are scrolling, and within ten you are grading vibes.

The fix is not a longer rubric document. It is a smaller prompt. Ask the candidate to ship a scoreboard: a script that grades their own submission against probes they wrote, in a form you can re-run on your machine without their laptop, their shell history, or their encouragement.

The prompt that makes grading possible

Keep the implementation small enough that it does not dominate the task. A rolling-window counter over a JSONL event stream is plenty: read {"t": <int seconds>, "key": "a"} lines, emit one line per event with allowed: true|false, honour --window and --limit. Two hours of work, maybe three.

Then add the second half of the prompt, which is the part that matters:

Ship scoreboard.py. It must run offline, take under 120 seconds, print a single JSON object to stdout, and exit non-zero if any probe fails. Include at least six probes. Write one probe for a case you are not sure you got right, and say in the handoff which one that is.

That last sentence is the whole interview. Anyone can write probes their code passes. Asking for the probe the candidate is least sure about tells you whether they know where their own edges are, and it gives you a concrete thing to talk about for the remaining thirty minutes.

One shape that works

A probe registry keeps this small. Each probe is a function returning (status, detail), and the runner does not care what language the implementation is in as long as it is invokable.

# scoreboard.py — illustrative sketch; run it before you trust it.
from __future__ import annotations
import json, pathlib, subprocess, sys, tempfile

PROBES: list[tuple[str, int, callable]] = []

def probe(name: str, weight: int = 1):
    def wrap(fn):
        PROBES.append((name, weight, fn))
        return fn
    return wrap

def run(stdin: str, window: int = 60, limit: int = 2):
    return subprocess.run(
        [sys.executable, "-m", "budgetq", "--window", str(window), "--limit", str(limit)],
        input=stdin, text=True, capture_output=True, timeout=20,
    )

@probe("boundary_exact_second", weight=2)
def boundary_exact_second(_tmp: pathlib.Path):
    feed = '{"t":0,"key":"a"}\n{"t":59,"key":"a"}\n{"t":60,"key":"a"}\n'
    p = run(feed)
    if p.returncode != 0:
        return "fail", f"exit {p.returncode}: {p.stderr.strip()[:200]}"
    got = [json.loads(l)["allowed"] for l in p.stdout.splitlines() if l.strip()]
    return ("pass", "") if got == [True, True, False] else ("fail", f"got {got}")

@probe("duplicate_event_ids_are_idempotent", weight=2)
def duplicate_event_ids_are_idempotent(_tmp: pathlib.Path):
    line = '{"t":10,"key":"a","id":"e1"}\n'
    p = run(line * 3)
    got = [json.loads(l)["allowed"] for l in p.stdout.splitlines() if l.strip()]
    return ("pass", "") if got == [True, False, False] else ("fail", f"got {got}")

def main() -> int:
    tmp = pathlib.Path(tempfile.mkdtemp())
    results, failed = [], 0
    for name, weight, fn in PROBES:
        try:
            status, detail = fn(tmp)
        except Exception as exc:
            status, detail = "error", f"{type(exc).__name__}: {exc}"
        results.append({"probe": name, "weight": weight, "status": status, "detail": detail})
        failed += weight if status != "pass" else 0
    total = sum(w for _, w, _ in PROBES)
    passed = sum(r["weight"] for r in results if r["status"] == "pass")
    print(json.dumps({"score": round(passed / total, 3), "results": results}, indent=2))
    return 0 if failed == 0 else 1

if __name__ == "__main__":
    raise SystemExit(main())

Note what the probe above actually asserts. It pins the meaning of the window boundary — is t=60 inside the window that started at t=0? — instead of re-asserting the --limit flag. Probes that restate the CLI are decoration. Probes that pin an ambiguous decision are the artifact you wanted.

The sealed probe

A scoreboard the candidate controls is still gameable, because a probe that always passes scores 1.0. Fix that with one file the candidate never sees. You write probes_sealed.py before sending the task, keep its hash, and require the runner to accept an external probe path:

def load_external(path: str) -> None:
    import hashlib, importlib.util
    spec = importlib.util.spec_from_file_location("sealed", path)
    mod = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(mod)  # registers through the same @probe decorator
    digest = hashlib.sha256(open(path, "rb").read()).hexdigest()[:12]
    print(f"loaded {len(mod.PROBES)} sealed probes sha256={digest}", file=sys.stderr)

Now the review is two commands, and the second one is yours:

cd takehome && python -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt
python scoreboard.py | tee run.candidate.json
python scoreboard.py --external probes_sealed.py; echo "sealed_exit=$?"

If the candidate's scoreboard passes theirs and fails yours, that is not a rejection. That is a forty-minute conversation with a scoreboard in the middle of it, which is more than most hiring loops ever produce.

Where the free tier earns its place

There is a fairness problem hiding in this prompt. You are asking candidates to spend their own time and, if they reach for a hosted model to draft probe ideas, their own money. Not every candidate has an API key with headroom, and the ones who do are not automatically the better engineers.

This is where the free model access and free server in the MonkeyCode open-source project become relevant rather than decorative. The operator describes a free tier that includes a token allowance (10 million at the time of writing) and a free server option, which is enough for a candidate to draft edge cases, run the harness against them, and leave behind something the reviewer can execute in the same environment. I would treat those figures as a snapshot and confirm the current terms on the project's site before you write them into a job posting.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A prompt to the free model that has actually been useful, kept short so it does not flood the context:

You are reviewing a specs document for a 60-second rolling-window
counter over JSONL events. List 12 boundary cases a reviewer would
test. For each: exact input lines and expected output. Skip anything
implied by the --limit flag alone.

Keep the model's output in probes_draft.md and hand-curate it into scoreboard.py. Shipping raw model output as your test suite is the fastest way to a scoreboard full of tautologies.

The failure modes you will actually see

Three of them, over and over. The first is the mirror suite: every probe constructs an input the implementation already handles, so the score is 1.0 and says nothing. The sealed probe catches this, and so does asking which probe they were unsure about — a candidate who cannot name one has not looked closely at their own work.

The second is the harness that only runs on the author's machine. Hard-coded absolute paths, a missing requirements.txt, a dependency on a local .env, a sleep(2) that assumes a warm cache. Time-box it: if pip install plus one run takes more than two minutes on a clean checkout, the scoreboard is broken, not slow.

The third is prose where JSON should be. A candidate who prints a friendly summary is writing a README with extra steps, and your review is back to vibes. The --external run and the exit code are the contract; everything else is commentary.

Who should not use this

If you cannot pre-run the sealed probe against your own reference solution, do not send the task — you will ship a prompt you cannot grade. If your take-home is a slice of real proprietary work, skip it entirely; a scoreboard does not fix asking someone to do free production work. And for junior roles, consider handing over the rubric and asking only for four probes against your reference implementation, since authoring a full harness measures tooling maturity more than it measures the skill you are hiring for.

What you get in return is a submission with a verdict attached. Not "I'm confident this is production ready" — a JSON object with a score, a list of probes, one of them failing on purpose, and a named uncertainty you can spend the interview on. That is a much better use of twenty minutes.

Comments

[Alex Shev]

A take-home is fairer when candidates can see the scoring model before they invest the time. I’d separate functional correctness, trade-off explanation, testing, and communication, with examples of what “good enough” means for each. It reduces guesswork and makes evaluator disagreement visible instead of retrofitting a rubric after submissions arrive.