The Take-Home Starts With a Red Log

You unzip the take-home. Tests pass. The README thanks you for your time. There is no failing log, no timestamp, no proof the candidate ever watched the bug happen.

That silence is the interview now. Models write tidy trees. A green zip is cheap. A red run on a real machine is not.

The last week on DEV was full of arguments about whether AI already codes better than most of us, and whether most “agents” are just if-statements in a nicer coat. Fine debates. They do not help you hire. You are not grading whether a model can emit a function. You are grading whether a human can refuse to ask the model until a test is actually red.

Here is a take-home packet you can send next week. It is a proposal, not a field study. Treat the code as a template you adapt, not as a scoreboard I ran against a hiring pipeline.

The prompt you send

Paste this as the assignment. Keep it short enough that a model cannot hide a full app inside a clarifying question.

You are fixing a billing helper. The helper lives in a remote session I provision. You may use a coding model. You may not paste a finished tree that was already green on your laptop.

invoices.py sums a CSV of charges. Refunds are stored as negative cents. The current code wraps every row in abs(), so refunds inflate the total.

You will be scored on order, not on prose. Run the tests and capture the failure in artifacts/red.log before you touch a model. Write or extend one test that names the refund case in plain language. Only then prompt a model. Save the exact prompt as artifacts/prompt.txt and the raw reply as artifacts/model.md. Apply a patch you understand. Run the tests again and save artifacts/green.log. Add artifacts/refuse.md: one unsafe patch the model suggested, or that it could have suggested, and why you would not apply it.

Ninety minutes. If the model is slow, say so in artifacts/notes.md. Do not silently skip the red run. I will not read style nits, extra features, or a second CSV parser.

That last sentence is the assignment doing real work. Candidates burn loops polishing names when you never asked for names.

The fixture

Ship a tiny tree. If the fixture looks like a product, the model will generate a product and you will learn nothing.

# invoices.py
import csv
from pathlib import Path

def total(path: str) -> int:
    n = 0
    with Path(path).open() as f:
        for row in csv.DictReader(f):
            n += abs(int(row["cents"]))
    return n

# test_invoices.py
from pathlib import Path
import invoices

FIXTURE = Path(__file__).parent / "sample.csv"

def test_charges_only():
    assert invoices.total(FIXTURE) == 1500

cents
1000
500
-200

The test is wrong on purpose. It encodes the bug. A candidate who “fixes” the expectation to stay at 1500 after a model comment has failed, even if pytest is green. You want that trap. Not a riddle. A check that they opened the CSV.

The gate

Most people will say they ran tests first. Do not take their word. Put a gate on the remote session and make the session enforce it.

# gate.py
from pathlib import Path
import sys, time, subprocess

ART = Path("artifacts")
ART.mkdir(exist_ok=True)
RED = ART / "red.log"
PROMPT = ART / "prompt.txt"
STAMP = ART / "model_allowed.at"

def run_tests(log: Path) -> int:
    p = subprocess.run(
        [sys.executable, "-m", "pytest", "-q"],
        capture_output=True,
        text=True,
    )
    log.write_text(p.stdout + p.stderr)
    return p.returncode

cmd = sys.argv[1] if len(sys.argv) > 1 else "help"

if cmd == "red":
    code = run_tests(RED)
    if code == 0:
        sys.exit("red.log is green. The gate wants a failure.")
    print("red captured")
elif cmd == "model":
    if not RED.exists() or "FAILED" not in RED.read_text():
        sys.exit("No red log. The model waits.")
    if not PROMPT.exists():
        sys.exit("Write artifacts/prompt.txt first.")
    STAMP.write_text(str(time.time()))
    print("model window open")
elif cmd == "green":
    if not STAMP.exists():
        sys.exit("Green before model is a skip, not a pass.")
    code = run_tests(ART / "green.log")
    if code != 0:
        sys.exit("still red")
    print("green captured")
else:
    print("usage: python gate.py red|model|green")

On the box they type this, in this order:

python -m pytest -q
python gate.py red
# edit test_invoices.py so the refund case is explicit
python gate.py model
# apply only the patch you accept
python gate.py green
ls -l artifacts

If they invert the commands, the gate yells. That yell is the interview. Think of the model as a contractor standing in the hallway. You do not hand it the keys until the alarm has already gone off.

A sample solution, labeled as a sample

This is an unexecuted example. Read it as a shape, not as a benchmark.

The candidate rewrites the test to expect 1300 and says why in the assertion name or a one-line comment: 1000 + 500 + (-200). They take abs() out of the production path and leave the CSV alone.

def total(path: str) -> int:
    n = 0
    with Path(path).open() as f:
        for row in csv.DictReader(f):
            n += int(row["cents"])
    return n

Their artifacts/prompt.txt is boring, which is good:

invoices.py uses abs() on cents. Refunds are negative.
test_invoices.py should expect 1300 for sample.csv.
Propose a minimal patch. Do not add new files.
Do not drop rows. Do not pull in pandas.

Their artifacts/refuse.md catches the “helpful” reply that wants to drop the refund row, coerce the column with abs, or switch the helper to a dataframe. They write one sentence: dropping rows changes revenue, it does not fix the sum.

You are scoring that sentence more than the patch. The patch is three lines. The refusal is the adult in the room.

Rubric

Keep four signals. If you add a fifth, you will argue about it in the debrief instead of hiring.

Signal	Pass	Fail
Red log is a real pytest failure	FAILED in red.log, mtime before the model stamp	Missing, empty, or already green
Test names the refund	Assertion or name mentions negative cents	They only renamed variables
Model waited	model_allowed.at after red	They pasted a patch with no stamp
Refusal is specific	Names a harmful alternative	“I would be careful” with no example

A candidate can fail the patch and still pass. Maybe they ran out of time after a clean red run and a sharp refuse note. That person understood the machine. The person with a perfect diff and no red log understood a chatbot.

Do not average the four cells into a single “AI skill” score. Order is a different muscle from taste. If you blend them, the green zip wins again.

Where the session lives

Do not ask candidates to buy an API key for a ninety-minute exercise. That filters for credit cards, not for judgment. You need a box that can run pytest and a model client that does not live on their laptop clock.

If you need a shared place where they can reach a coding model and a remote box without standing up their own cluster, MonkeyCode's free model access and free server option can host that session. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Plug gate.py into that workspace the same way you would plug it into any other SSH box. The protocol is the product. The vendor is furniture.

Freeze the gate, not the vibes. If red.log can be written after the fact, you are grading theater. If one candidate’s model is slower, that belongs in notes.md. It does not get to skip red.

How people fail this

The confident skip is the common one. They run pytest locally, see the bug, ask a model on their laptop, upload a green tree, then try to forge red.log by hand. The forge is usually too clean. Real pytest noise has session headers and a trailing 1 failed. A handwritten log looks like a haiku.

The second failure is the overfit test. They change the expected total to 1500 and leave abs() in place. Green. Wrong. The gate does not catch that. You do, in five seconds, because you can add.

The third is the essay. They write four paragraphs about agent loops and never open the CSV. You asked for a red log. They sent a keynote.

The fourth is the unsafe apply. The model suggests eval on a row, or reading a sidecar file from /etc. They apply it because the tests still pass. refuse.md is empty. That is a no, even if the sum is right. A take-home that never asks for a refusal trains people to accept the first patch that quiets pytest.

Watch timestamps. A green.log that predates red.log is not a mystery. It is a candidate who worked backwards from a finished answer. You already know that pattern from take-homes that never involved a model. The model only made it faster.

Who should not use this

If you are hiring for a role that will never touch tests, do not send this. You will punish people for a skill you do not pay for.

If your legal team forbids any external model, do not pretend the gate is optional. Either you provide the model on your server, or you rewrite the exercise as a paper patch with the same red-before-diff rule.

If you need a two-day product slice, this packet will feel like a toy. It is a toy. It is a toy that tells you whether someone can wait.

Juniors who have never seen pytest can still pass if you add one extra line to the prompt: red means the runner printed FAILED. Do not hide the protocol behind jargon. The point is the order, not the tooling resume.

Skip it for pairing loops you can already watch. If you will sit with them for an hour, you do not need a stamp file. You can see when they reach for the model. This packet is for the async take-home, where you only meet the zip.

After you collect the zips

Open red.log first. Then the stamp. Then the refuse note. Then, if you still care, the diff.

You will throw some green zips away. That will feel rude the first time. It is ruder to hire a person who cannot tell you when the test was red.

Hold the model until the log exists. Everything after that is commentary.