The Take-Home Where the Model Is Wrong on Purpose

A candidate shipped his take-home on a Tuesday night. Tests green, README tidy, forty-minute turnaround — the fastest submission of the cohort. Then the payments stub logged two charges for one order. He had done exactly what the model told him to do, which is the whole reason this take-home exists.

You can feel the same anxiety in the DEV feed this week. Two of the bigger discussions land on it from opposite sides: "AI Got Better While I Was Away. Software Didn't." and "Super-Intelligent Yes-Men." I'm not citing either as evidence — they're topic signals, not data. But they name the thing interviewers keep asking about: how do you grade a candidate when the code arrived faster than the thinking?

Banning the assistant doesn't work. You can't enforce it, and you'd be testing a skill nobody brings to the job. What you can do is hand out a task the model gets almost right, then grade whether the candidate notices. Judgment, not keystrokes.

The prompt

Keep it small enough to finish in an evening, and sharp enough that one wrong decision costs real money.

TASK: Implement a payment client for the gateway stub in ./gateway.

Constraints
- Calls may fail transiently (503). Retry up to 3 times with backoff.
- The gateway may receive a request twice. A single order must never
  result in two charges.
- Assume the process can be restarted between attempts.

Deliverables
- checkout.mjs exporting chargeWithRetry(amount)
- A short note: what you verified, and what you could not verify
- No new dependencies

You may use any assistant you like. Tell us which, and which parts
of the diff came from it.

That last line does more work than it looks like. It makes the model's help visible, so disclosure becomes normal instead of a confession. Candidates who hide it aren't punished for cheating; they're punished for handing you an unreviewable diff.

The harness

Here's the smallest thing that catches the seeded defect: a gateway that always fails the first attempt, and records every key it sees.

// fake-gateway.mjs — run with: node fake-gateway.mjs
import { createServer } from "node:http";

const seen = [];

const server = createServer((req, res) => {
  let body = "";
  req.on("data", (chunk) => (body += chunk));
  req.on("end", () => {
    const { idempotencyKey, amount } = JSON.parse(body || "{}");
    seen.push({ key: idempotencyKey, amount });

    const isFirstAttempt = seen.length === 1;
    res.writeHead(isFirstAttempt ? 503 : 200, {
      "content-type": "application/json",
    });
    res.end(JSON.stringify({ ok: !isFirstAttempt }));

    if (seen.length === 2) {
      const distinct = new Set(seen.map((s) => s.key));
      console.log(JSON.stringify({
        charges: seen.length,
        distinctKeys: distinct.size,
      }));
      server.close();
    }
  });
});

server.listen(8099, () => console.log("gateway listening on :8099"));

The draft most assistants hand you looks like this. Read it once and it seems fine; the bug is one line's scope.

// checkout.mjs — the version most assistants produce
export async function chargeWithRetry(amount, attempts = 3) {
  for (let i = 0; i < attempts; i++) {
    const res = await fetch("http://localhost:8099/charge", {
      method: "POST",
      headers: { "content-type": "application/json" },
      body: JSON.stringify({
        amount,
        idempotencyKey: crypto.randomUUID(), // fresh key per attempt
      }),
    });
    if (res.ok) return res.json();
    await new Promise((r) => setTimeout(r, 50 * (i + 1)));
  }
  throw new Error("charge failed after " + attempts + " attempts");
}

Run both files and the stub prints {"charges":2,"distinctKeys":2}. Two charges, two keys, one order. The gateway behaved correctly the whole time — it dedupes what you tell it to dedupe, and this client told it these were two different payments.

The fix is boring, which is the point. Hoist the key out of the loop so every attempt carries the same one:

const idempotencyKey = crypto.randomUUID(); // one key per logical charge
for (let i = 0; i < attempts; i++) { /* ... */ }

Now the stub prints {"charges":2,"distinctKeys":1}, and a real gateway would collapse the second attempt into the first. Strong candidates go one step further and persist the key before the first call, because "assume the process can be restarted" means the in-memory key dies with the process — and the retry after that restart becomes a second charge.

What you're grading

Pin this next to the submissions. The left column is behavior; the rest are the tiers you actually score.

Signal	Weak	Acceptable	Strong
Found the defect	Did not see it	Found it by reading the diff	Found it because a test went red
Key scope	Per attempt	Per call	Per call, persisted before the request
Evidence	"It passes"	Ran the stub once	Before/after runs with raw output
Explanation	Restates the diff	Names idempotency	Explains the restart gap unprompted
Blast radius	Ignores it	Mentions duplicates	Names what else is unsafe: timeouts, partial writes

Notice that "found it by reading the diff" and "found it because a test went red" score differently. The second one is the skill you're hiring for.

Failure modes worth pre-committing to

Candidates who fix the symptom. They wrap the loop in a try/catch and swallow the 503, so the stub never retries: one charge, zero retries, and a payment silently dropped during an outage. Green harness, broken product.

Candidates who delete the model's work. A full rewrite that also removes the backoff looks decisive and scores low. You asked for a judgment call, and they answered with a repo-wide replace.

Reviewers who leak the rubric. If the prompt mentions idempotency keys anywhere, the task collapses into a vocabulary quiz. Say "never two charges" and let them find the mechanism.

Reviewers who grade the diff and skip the note. The note is where the restart gap surfaces. If your process only reads code, you're grading half the submission.

Where MonkeyCode fits

Two parts of this task need compute you can't assume a candidate has. The first is the model draft: they need an assistant to produce the subtle version honestly, and someone without paid API credits shouldn't be pushed into writing a weaker baseline just to save money. The second is the harness: a laptop with a mismatched Node version, or an ARM machine under a locked-down corporate image, can turn a five-minute task into a lost evening.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source coding assistant, and the operator lists free model access plus a free server option, with the free tier described as including a 10M-token allowance. Treat that as a claim to verify on the current pricing page before you run a whole cohort on it — free tiers move, and I'd rather you check than trust a blog post. In practice the free server is the more interesting half here: when every candidate runs the stub on the same pinned box, distinctKeys means the same thing for all of them, and your rubric stops accidentally measuring their laptop.

If you want to dry-run the model-draft half without buying credits first, that's the cheapest path I know.

Limitations and who should not use this

This task tests judgment about machine output. It does not test systems design, data modeling, or anything that needs a week of context, so don't use it as your only signal for a senior backend hire.

The harness above is a template, not a validated benchmark. I kept it to the parts you can reason about: one endpoint, one failure mode, one assertion. Your gateway semantics may differ, and real idempotency windows are time-bounded, which this stub cheerfully ignores.

Skip the format entirely if your team can't commit to reading the note, or if you plan to run a hundred candidates past one reviewer. Slow, careful reading is the whole mechanism. Automate the rubric and you'll hire the people who guessed your assertions.