A Take-Home Packet That Refuses to Trust a Green Exit Code

A green pytest exit code is not proof that generated tests exercised the code under review. Interview loops that grade from a chat transcript, or from process status alone, will accept empty suites. This take-home packet scores collection counts, JUnit artifacts, import graphs, and one mutation kill instead. The domain is half-open interval merging, which is small enough that the rubric can stay strict and mechanical.

Why a zero status is the wrong grade

Coding agents often summarize a run instead of leaving machine-checkable files. A model can emit a calm “all tests passed” after writing assert True, after skipping every case, or after placing tests beside the wrong package. Pytest still exits zero when it collects zero items from an empty directory, and many wrappers treat that status as success.

The failure is ordinary rather than exotic. Generated work can look complete while the verification loop never imported the module under review. A hiring process that cannot tell those cases apart will rank confident narration above working software. Recent public debate about evaluation catching up with generated code points at the same gap, without requiring any particular product claim.

The assignment in one page

The candidate receives a tiny booking helper and a nearly empty test tree. The task is to merge half-open integer intervals correctly and to prove that merge with tests the grader can re-run. The prompt below is meant to be pasted into an agent session or handed to a human as a timed take-home.

Prompt for the candidate

You are editing a small Python 3.12 package named slotmerge.

Fix slotmerge/intervals.py so that merge_intervals(spans) returns a
sorted list of half-open [start, end) integer spans. Empty spans
(end <= start) must be dropped. Overlapping spans must merge.
Touching spans such as [0, 2) and [2, 5) must also merge to [0, 5).

Add tests under tests/ that pytest will collect. Constraints:

- Tests must import slotmerge.intervals and call merge_intervals.
- Do not mock merge_intervals.
- Do not skip tests in order to obtain a green run.
- Do not add assert True placeholders.
- Include at least four cases, one of which is a touching pair.
- After your change, python -m pytest -q tests must collect those
  tests and fail if merge_intervals stops merging touching spans.

Return a short report that includes the pytest JUnit XML path, the
collected count, and the commands you actually ran. Do not claim a
pass without that XML file.

Starter tree

slotmerge/
  pyproject.toml
  slotmerge/
    __init__.py
    intervals.py
  tests/
    conftest.py

Starter intervals.py is intentionally wrong. It drops empty spans, yet it treats touching ranges as separate rooms.

from __future__ import annotations


def merge_intervals(spans: list[tuple[int, int]]) -> list[tuple[int, int]]:
    cleaned = [(s, e) for s, e in spans if e > s]
    if not cleaned:
        return []
    cleaned.sort(key=lambda se: (se[0], se[1]))
    out = [cleaned[0]]
    for start, end in cleaned[1:]:
        last_s, last_e = out[-1]
        # BUG: touching half-open spans should merge, overlap only is not enough.
        if start < last_e:
            out[-1] = (last_s, max(last_e, end))
        else:
            out.append((start, end))
    return out

Starter tests/conftest.py is empty on purpose. Many agents add test_intervals.py next to the source file, or they write a file that never imports slotmerge.

A minimal pyproject.toml keeps collection deterministic for the grader.

[project]
name = "slotmerge"
version = "0.1.0"
requires-python = ">=3.12"

[project.optional-dependencies]
test = ["pytest==8.3.3"]

[tool.pytest.ini_options]
testpaths = ["tests"]
pythonpath = ["."]

What the grader actually checks

The hidden harness never treats process status as the grade. It runs four steps in order and stores each artifact on disk. Interviewers should read those files rather than the assistant’s closing paragraph.

1. Collection — pytest --collect-only -q tests must report at least four items.
2. JUnit — pytest --junitxml=artifacts/junit.xml tests must create that file and list those cases.
3. Import graph — every tests/test_*.py file must import slotmerge.intervals in the AST.
4. Mutation kill — the harness copies the tree, changes the overlap test from start <= last_e to start < last_e, and expects pytest to fail.

A candidate who only prints “passed” without XML fails step 2. A candidate who writes tests beside the source file, outside tests/, fails step 1. A candidate who mocks merge_intervals fails step 3. A candidate whose tests never include a touching pair fails step 4.

Reference grader (local, labeled)

The copy-and-mutate loop below is a reference implementation for a throwaway tree. Operators should run it in CI or on an isolated runner, not by trusting comments in a chat log.

# grader.py — reference implementation for local use
from __future__ import annotations

import ast
import pathlib
import shutil
import subprocess
import sys
import tempfile
import xml.etree.ElementTree as ET

ROOT = pathlib.Path(__file__).resolve().parent
XML = ROOT / "artifacts" / "junit.xml"


def run(args: list[str], cwd: pathlib.Path) -> subprocess.CompletedProcess[str]:
    return subprocess.run(args, cwd=cwd, text=True, capture_output=True, check=False)


def collected_count(cwd: pathlib.Path) -> int:
    proc = run([sys.executable, "-m", "pytest", "--collect-only", "-q", "tests"], cwd)
    for line in reversed(proc.stdout.splitlines()):
        parts = line.strip().split()
        if parts and parts[0].isdigit() and "collected" in line:
            return int(parts[0])
    return 0


def tests_import_module(cwd: pathlib.Path) -> bool:
    for path in (cwd / "tests").glob("test_*.py"):
        tree = ast.parse(path.read_text(encoding="utf-8"))
        for node in ast.walk(tree):
            if isinstance(node, ast.ImportFrom) and (node.module or "").startswith("slotmerge"):
                return True
            if isinstance(node, ast.Import):
                for alias in node.names:
                    if alias.name.startswith("slotmerge"):
                        return True
    return False


def write_junit(cwd: pathlib.Path) -> None:
    (cwd / "artifacts").mkdir(exist_ok=True)
    run(
        [sys.executable, "-m", "pytest", f"--junitxml={cwd / 'artifacts' / 'junit.xml'}", "-q", "tests"],
        cwd,
    )


def mutation_killed(cwd: pathlib.Path) -> bool:
    with tempfile.TemporaryDirectory() as tmp:
        dest = pathlib.Path(tmp) / "slotmerge"
        shutil.copytree(cwd, dest, ignore=shutil.ignore_patterns(".venv", "artifacts"))
        target = dest / "slotmerge" / "intervals.py"
        text = target.read_text(encoding="utf-8")
        if "start <= last_e" not in text:
            return False
        target.write_text(text.replace("start <= last_e", "start < last_e", 1), encoding="utf-8")
        proc = run([sys.executable, "-m", "pytest", "-q", "tests"], dest)
        return proc.returncode != 0


def main() -> int:
    XML.parent.mkdir(exist_ok=True)
    n = collected_count(ROOT)
    write_junit(ROOT)
    if n < 4:
        print("FAIL: collected fewer than four tests")
        return 1
    if not XML.exists():
        print("FAIL: missing JUnit XML")
        return 1
    cases = ET.parse(XML).findall(".//testcase")
    if len(cases) < 4:
        print("FAIL: JUnit listed fewer than four cases")
        return 1
    if not tests_import_module(ROOT):
        print("FAIL: tests never import slotmerge")
        return 1
    if not mutation_killed(ROOT):
        print("FAIL: touching-span mutant still green")
        return 1
    print("PASS: collection, XML, imports, and mutation kill")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Rubric

Score the packet out of twelve points. Passing is nine or higher, and the mutation kill is mandatory. Partial credit on style comments should not rescue a missing XML file.

Signal	Points	Zero if
Four or more collected tests	2	Zero items, or skipped-only collection
JUnit XML matches collection	2	Chat claims a pass and the file is missing
Tests import and call merge_intervals	2	Mocked symbol, dynamic hide, or no import
Touching pair is a first-class case	3	Only strict overlap, or only disjoint spans
Mutant that uses < instead of <= fails	3	Suite stays green after the mutation

Half-open touching spans merge with a single comparison after a stable sort.

from __future__ import annotations


def merge_intervals(spans: list[tuple[int, int]]) -> list[tuple[int, int]]:
    cleaned = [(s, e) for s, e in spans if e > s]
    if not cleaned:
        return []
    cleaned.sort(key=lambda se: (se[0], se[1]))
    out = [cleaned[0]]
    for start, end in cleaned[1:]:
        last_s, last_e = out[-1]
        if start <= last_e:
            out[-1] = (last_s, max(last_e, end))
        else:
            out.append((start, end))
    return out

A focused test file makes the touching contract explicit instead of re-encoding the production bug.

from slotmerge.intervals import merge_intervals


def test_touching_spans_merge() -> None:
    assert merge_intervals([(0, 2), (2, 5)]) == [(0, 5)]


def test_overlap_extends_end() -> None:
    assert merge_intervals([(0, 4), (2, 5)]) == [(0, 5)]


def test_empty_span_dropped() -> None:
    assert merge_intervals([(3, 3), (1, 2)]) == [(1, 2)]


def test_disjoint_preserved_order() -> None:
    assert merge_intervals([(10, 12), (0, 1)]) == [(0, 1), (10, 12)]

Common failure modes

Agents and rushed humans fail this packet in a small number of repeatable ways. Interviewers can map each failure to a process smell rather than to a personality judgment.

• Empty collection. Pytest is invoked on the repo root, finds no test_*.py, and still exits zero.
• Wrong tree. Tests land under slotmerge/test_intervals.py, which the configured testpaths never collect.
• Tautology. The test recomputes the same < rule and compares the function to itself.
• Skipped green. @pytest.mark.skip wraps every failing case so the status becomes zero.
• Over-mock. merge_intervals is patched to return expected tuples, so the mutant never dies.
• Narrative proof. The report says “all tests passed” with no XML path and no command log.
• Overlap-only coverage. Cases never include [0, 2) against [2, 5), so the off-by-one survives.

Each of those still produces a fluent summary in a chat window. That is why the rubric ignores the summary and reads JUnit plus the mutant.

Running the packet on an isolated runner

A take-home like this only works if the grader actually executes pytest on a clean tree. Local laptops work. Ephemeral servers also work, and they keep candidate runs isolated from interviewer files and secrets.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is a coding assistant that, per the operator of this article, offers free model access and a free server option. Those two availability facts are the only product claims made here. They matter for this packet because the mutation job and the JUnit parse should run on a machine the candidate does not control, not inside a model transcript.

A practical loop looks like the following commands. Clone the starter, paste the prompt, let the assistant edit, then grade with commands the interviewer owns.

python -m venv .venv
source .venv/bin/activate
pip install -e ".[test]"
mkdir -p artifacts
python -m pytest --collect-only -q tests
python -m pytest --junitxml=artifacts/junit.xml -q tests
python grader.py

If the assistant is connected to a free server, the same commands run away from the interviewer’s laptop and leave XML under artifacts/. The interviewer still reads the XML, not the chat. Model names, hardware sizes, quotas, duration, and uptime are omitted because they are not verified in this article.

Readers who already have pytest and a spare VM do not need another runner. MonkeyCode is relevant only when a team wants a disposable server plus model access without standing up its own inference stack. One optional next step is to load this packet onto that free server and compare JUnit files across two attempts.

Limitations and who should skip this

This packet measures a narrow contract: interval merge, collection, imports, and one mutant. It does not measure API design, calendar time zones, or production scheduling rules. Half-open integer minutes are a chosen convention, and real booking systems also deal with time zones, inclusive ends, and inclusive-exclusive mismatches at day boundaries.

Do not use this as a sole hiring bar. Do not run candidate code on a server that holds secrets, production credentials, or unrelated customer data. Do not treat a killed mutant as proof of overflow handling, huge span lists, or concurrent mutation of the input list. The AST import check is syntactic and will miss dynamic imports constructed from strings.

Teams that already require CI logs as the only accepted proof will find the packet redundant. Teams that currently paste agent chat into a scorecard will not. The method also fails closed if pytest configuration diverges between the candidate tree and the grader tree, so freeze the pytest version in pyproject.toml.

Closing notes

Green exit codes are cheap to produce and expensive to believe. Collection counts, JUnit files, import graphs, and mutation kills are slightly less cheap and much harder to fake in a short take-home. This packet keeps the domain small so the rubric can stay strict. Interviewers who adopt it should version the starter tree and pin pytest, so yesterday’s empty pass cannot become tomorrow’s hidden collection path.