The Rejection File: A Take-Home That Grades the Patch the Candidate Refused to Ship

A take-home that grades only the final diff measures output volume; the stronger hiring signal sits in the proposals a candidate refused to ship. The Rejection File is a small exercise that asks for a working patch plus an auditable record of at least one model proposal the candidate rejected, together with a command that proves the refusal was correct. It fits in two to three hours, runs on free-tier infrastructure, and separates engineers who verify from engineers who relay.

The format responds to a DEV thread this week arguing that engineers should not become a proxy between a requester and an AI. That complaint is really about unverified relay: a proposal arrives, a patch leaves, and nobody checks the middle step. A rejection file makes the middle step the deliverable, so reviewers can inspect judgment instead of guessing at it.

The signal a diff-only take-home cannot see

Most take-homes reward candidates for producing code that passes hidden tests, which is a narrow measurement of one skill. A model can generate a passing patch from a decent prompt, so the passing patch no longer distinguishes much. What still distinguishes candidates is whether they noticed that two of the four proposals were quietly wrong.

Rejecting a proposal is cheap to claim and expensive to prove, which is exactly why it makes a good artifact. Anyone can write "I rejected the broad exception handler because it hides errors" in a README. Requiring a repro command that fails for the rejected proposal turns the claim into evidence a reviewer can replay in one terminal command.

The prompt, written for the candidate

The following text can be handed over verbatim; it assumes an isolated fixture repository with a small, real defect.

1. Fix the defect in src/. Any approach is acceptable as long as the existing test suite stays green.
2. Use an AI assistant for at least two candidate proposals. Record each proposal in rejections.json using the schema below.
3. Reject at least one proposal and supply a repro command that fails when your rejection is correct.
4. Keep the accepted diff under roughly forty changed lines unless the repository forces more; explain any overrun in the handoff note.
5. Write a half-page handoff note covering what you verified, what you assumed, and what you would check next.

The constraints matter more than the defect. Candidates who burn the whole repository into context tend to produce a plausible patch plus a rejection file full of unverifiable prose, which the rubric scores accordingly.

The rubric

Criterion	Weight	What earns full credit
Rejection evidence	30	Every rejection has a repro command that actually fails
Patch correctness	20	Full suite green, including tests the candidate did not write
Reproduction quality	15	One command, no manual setup steps, deterministic output
Scope discipline	15	Diff stays near the defect; unrelated refactors are absent
Budget discipline	10	Token spend is recorded and re-prompting loops are explained
Handoff note	10	Separates verified facts from assumptions without hedging

The weighting is deliberate. A candidate with a mediocre patch and three credible rejections generally scores higher than one with a clean patch and a rejection file that cannot be replayed, because the second candidate produced no inspectable judgment.

The artifact: a rejection file that can be audited

The schema is intentionally boring so that a script can check it.

{
  "acceptance": {
    "repro": "pytest -q tests/test_resume.py::test_truncated_stream"
  },
  "rejections": [
    {
      "id": "reject-stream-close",
      "proposal": "Close the stream inside the except branch and return None",
      "repro": "pytest -q tests/test_resume.py::test_truncated_stream",
      "expect_failure": true,
      "decision": "rejected: callers cannot distinguish truncation from empty input"
    },
    {
      "id": "reject-broad-except",
      "proposal": "Wrap the consumer loop in except Exception and retry once",
      "repro": "pytest -q tests/test_cancel.py::test_cancel_mid_stream",
      "expect_failure": true,
      "decision": "rejected: converts a loud failure into an invisible stall"
    }
  ]
}

The audit script below is a reference sketch rather than a benchmarked tool, and it should be run against your own fixture before it grades anyone. It replays each repro inside the candidate's working copy and checks that the exit code matches the claim, then confirms that the accepted patch passes its own repro.

# rejection_audit.py — replay every rejection and confirm the claim holds
import json, pathlib, subprocess, sys

WORK = pathlib.Path("work")

def run(cmd: str, timeout: int = 120) -> int:
    proc = subprocess.run(cmd, shell=True, cwd=WORK, timeout=timeout,
                          capture_output=True, text=True)
    return proc.returncode

def audit(path: str) -> int:
    data = json.loads(pathlib.Path(path).read_text())
    bad = 0
    for entry in data["rejections"]:
        code = run(entry["repro"])
        credible = (code != 0) == entry["expect_failure"]
        print(f'{entry["id"]:<26} exit={code:<3} credible={credible}')
        bad += 0 if credible else 1
    accepted = run(data["acceptance"]["repro"])
    print(f'{"accepted patch".ljust(26)} exit={accepted:<3} credible={accepted == 0}')
    return bad + (accepted != 0)

if __name__ == "__main__":
    sys.exit(1 if audit(sys.argv[1]) else 0)

A non-zero exit from the auditor is not a rejection of the candidate; it is a signal for the reviewer to open the named entry and read the reasoning. The script checks whether evidence exists, not whether the reasoning is sound.

Running the exercise without paying per candidate

The environment cost is the part that usually kills custom take-homes, because per-candidate model spend and hosted runners add up across a hiring loop. MonkeyCode's free model access and free server option make a dry run practical: the reviewer keeps the fixture repository and the audit script, and candidates get a working environment without a procurement conversation.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Availability details come from the operator rather than from independent testing: free model access, a free server option, and a free token allotment quoted at ten million tokens at the time of writing. Those terms can change, so confirm the current limits before you put them in a candidate brief. The claim that matters for this exercise is narrower and easy to check yourself: a candidate can complete a two-hour patch-and-reject loop without a paid account.

git clone <your-fixture-repo> work
cd work && python -m venv .venv && . .venv/bin/activate
pip install -r requirements-dev.txt
python ../rejection_audit.py ../rejections.json

A worked sample

A strong submission produces output where every line is credible, and one weaker submission produces output where a single line is not.

$ python rejection_audit.py rejections.json
reject-stream-close        exit=1   credible=True
reject-broad-except        exit=1   credible=True
reject-retry-forever       exit=0   credible=False
accepted patch             exit=0   credible=True

The third entry is the interesting one. The candidate claimed to reject an infinite retry loop, but the supplied repro exits cleanly, so the rejection has no supporting evidence. In practice this is usually rejection theater: a proposal the candidate never seriously considered, documented after the fact to satisfy the prompt.

Reviewers should read the failing entry before scoring, because a credible rejection and an honest admission of a wrong repro look similar in the terminal. A candidate who writes "this repro passes and I no longer believe the rejection holds" is demonstrating better judgment than one who fabricates a failing command.

Common failure modes

1. Rejection theater. The rejected proposal is trivial, such as removing a comment, and the repro proves nothing about the defect.
2. Unreplayable evidence. The repro depends on a local file, an environment variable, or a manual step that the reviewer cannot reproduce.
3. Silent scope creep. A four-line defect becomes a two-hundred-line refactor, and the rejection file never explains why the extra surface was necessary.
4. Passing-by-accident patches. The suite is green because the candidate edited a test rather than the source, which the auditor catches only if fixtures are read-only.
5. Context dumping. Every re-prompt includes the entire repository, the budget is exhausted early, and the remaining decisions are made without evidence.
6. Narrated verification. The handoff note says "verified" without naming the command, the test, or the observed output that supports the word.

Mitigations are cheap: make fixture tests read-only in CI, cap the token allotment per candidate, and require one repro per rejection with no free-text substitute.

Limitations and who should skip this

The format needs a fixture repository with a genuine, small, test-detectable defect, and building that fixture takes a reviewer most of a day. Teams without a sandboxed runner should not adopt it, because replaying candidate commands on a shared host is a security problem rather than a convenience problem.

It also adds review time, roughly forty-five minutes per submission when the rejection file is read properly. Organizations that need same-day decisions, or roles where the work is not test-driven, get little from this exercise and should use a shorter structured interview instead. Finally, anyone who cannot confirm the current free-tier terms should describe the environment generically in the brief rather than quoting numbers that may have moved.

The trade is straightforward: a diff-only take-home measures what a candidate shipped, while a rejection file measures what a candidate was willing to stop. If you want to dry-run the harness before committing to it, run one internal submission through MonkeyCode's free server option and see whether the audit script survives your own repository first.