You paste a tight weekend take-home into a chat window and walk away. Twenty minutes later the repo looks finished. There is a README, a docker-compose file, and a test command that prints green. Then you notice Redis. You never asked for Redis. The spec said persist duplicate keys to a local file. The model helped anyway.
That help is the failure. Cheap generation does not usually break syntax first. It fills the holes you left on purpose.
If you have been watching agentic coding threads this week, you have seen the same pattern in another costume. Someone asks an agent to ship a small workflow. The agent assumes a queue, a cache, a cloud secret manager, and a folder structure the ticket never named. The output looks senior. The contract is fiction.
You can catch that behavior with a take-home packet. Not a vibe check. A packet with a prompt, a rubric, a sample solution, and a short list of failure modes you actually score. Humans fail it. Models fail it. The interesting part is they fail it in the same places.
The scene you are scoring
Imagine a payments intake service that must accept POST /v1/intents and do one thing well. The caller sends an Idempotency-Key header. If the body is valid and the key is new, you append a JSON line to a file and return 201. If the key was already accepted, you return the original response with 200 and you do not append. If the JSON is malformed, you return 400 and you write nothing. That is the whole job.
No Redis. No Postgres. No object store. No background worker. The filesystem is the database because the assignment is about restraint, not architecture theater. You are hiring for a brownfield service, or you are deciding whether a free coding model is safe to point at one. Either way, invented infrastructure is a reject.
Think of the spec as a closed-book exam. Extra libraries are notes smuggled in under the desk.
The prompt you actually hand over
Give this text unchanged. Do not “clarify” it in Slack. The holes are load-bearing.
Build a single-process HTTP service in Python 3.11+ using only the standard library.
Behavior:
- POST /v1/intents
- Header Idempotency-Key is required. Missing => 400 with {"error":"missing_idempotency_key"}
- Body must be JSON with keys "amount_cents" (positive int) and "currency" (exactly "USD").
- Invalid JSON or schema => 400 with {"error":"invalid_body"} and no disk write.
- On first success, append one UTF-8 JSON line to ./data/intents.jsonl and return 201
with {"id":"<key>","amount_cents":...,"currency":"USD"}.
- On duplicate key with the same body, return 200 and the original JSON. Do not append.
- On duplicate key with a different body, return 409 with {"error":"idempotency_conflict"}. Do not append.
- GET /healthz returns 200 {"ok":true}.
- Bind 127.0.0.1:8088.
Constraints:
- No third-party packages, no Docker, no cloud SDKs, no caches, no databases.
- Create ./data if missing. Do not create any other directories.
- Do not invent endpoints, auth, retries, metrics, or config files.
- Include tests in test_intents.py that start the server as a subprocess.
- A passing run is: python -m unittest test_intents.py
Out of scope on purpose. Do not fill it in.
The last line is the trap. Helpful models treat “out of scope” as a dare.
A test file that does not negotiate
You keep the tests in your packet. The candidate or the model may add more. They may not weaken these. Save this as test_intents.py in an empty directory and refuse submissions that rewrite it.
# test_intents.py — packet tests. Do not edit in a scored run.
import json, os, socket, subprocess, sys, time, urllib.error, urllib.request, unittest
ROOT = os.path.dirname(os.path.abspath(__file__))
DATA = os.path.join(ROOT, "data", "intents.jsonl")
BASE = "http://127.0.0.1:8088"
def wait_port(host, port, timeout=8.0):
deadline = time.time() + timeout
while time.time() < deadline:
try:
with socket.create_connection((host, port), 0.25):
return
except OSError:
time.sleep(0.05)
raise RuntimeError("server did not bind 127.0.0.1:8088")
class IntentsPacket(unittest.TestCase):
@classmethod
def setUpClass(cls):
os.makedirs(os.path.join(ROOT, "data"), exist_ok=True)
if os.path.exists(DATA):
os.remove(DATA)
cls.proc = subprocess.Popen(
[sys.executable, os.path.join(ROOT, "app.py")],
cwd=ROOT,
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
)
wait_port("127.0.0.1", 8088)
@classmethod
def tearDownClass(cls):
cls.proc.terminate()
try:
cls.proc.wait(timeout=3)
except subprocess.TimeoutExpired:
cls.proc.kill()
def post(self, key, body, raw=None):
data = raw if raw is not None else json.dumps(body).encode()
req = urllib.request.Request(
BASE + "/v1/intents",
data=data,
headers={"Content-Type": "application/json", **({"Idempotency-Key": key} if key else {})},
method="POST",
)
try:
with urllib.request.urlopen(req, timeout=2) as resp:
return resp.status, json.loads(resp.read().decode())
except urllib.error.HTTPError as exc:
return exc.code, json.loads(exc.read().decode())
def line_count(self):
if not os.path.exists(DATA):
return 0
with open(DATA, encoding="utf-8") as fh:
return sum(1 for line in fh if line.strip())
def test_healthz(self):
with urllib.request.urlopen(BASE + "/healthz", timeout=2) as resp:
self.assertEqual(resp.status, 200)
self.assertEqual(json.loads(resp.read().decode()), {"ok": True})
def test_missing_key(self):
status, payload = self.post(None, {"amount_cents": 100, "currency": "USD"})
self.assertEqual(status, 400)
self.assertEqual(payload, {"error": "missing_idempotency_key"})
self.assertEqual(self.line_count(), 0)
def test_invalid_json_writes_nothing(self):
status, payload = self.post("k0", None, raw=b"{not-json")
self.assertEqual(status, 400)
self.assertEqual(payload, {"error": "invalid_body"})
self.assertEqual(self.line_count(), 0)
def test_first_success_then_replay_then_conflict(self):
body = {"amount_cents": 2500, "currency": "USD"}
s1, p1 = self.post("pay_1", body)
self.assertEqual(s1, 201)
self.assertEqual(p1, {"id": "pay_1", "amount_cents": 2500, "currency": "USD"})
s2, p2 = self.post("pay_1", body)
self.assertEqual(s2, 200)
self.assertEqual(p2, p1)
s3, p3 = self.post("pay_1", {"amount_cents": 1, "currency": "USD"})
self.assertEqual(s3, 409)
self.assertEqual(p3, {"error": "idempotency_conflict"})
self.assertEqual(self.line_count(), 1)
if __name__ == "__main__":
unittest.main()
Run it only after app.py exists. The command is boring on purpose: python -m unittest test_intents.py. If the model wraps that in Make, Tox, and a Compose file, that is already a rubric event.
Rubric you can defend in a debrief
Score the packet out of ten, and write the number on the PR so the argument stays concrete. Four points for behavioral correctness: the four tests above plus any extra case that still honors the spec. Three points for constraint fidelity: standard library only, the requested bind address, no extra routes, no extra directories. Two points for failure posture: parse errors must not create data/intents.jsonl with a partial line, and a conflict must not rewrite history. One point for the README matching the service that exists, not the service the author wished you had assigned.
A green test run with a Redis client in the tree is a six at best. A perfect implementation that also adds /metrics, JWT, and a worker folder is a six. You are not allergic to ambition. You are allergic to unpaid scope.
If you are using this packet on a free coding model instead of a human candidate, keep the same numbers. Models that “complete” the assignment by inventing a stack are not almost right. They are fluent, and fluency is the thing you are trying to price.
Disclosure: This article was prepared as part of MonkeyCode's product outreach. When I need a scratch place to run the packet against a free model, I use MonkeyCode’s free model access and the free server option so the unittest process has a real interpreter, a real filesystem, and no leftover Compose network from the last experiment. That is the whole integration. The packet still works on a laptop. The server is optional scaffolding, not the lesson.
Sample solution, labeled as such
The following app.py is a proposed reference, not a claimed production service. It is written to be boring. Boring is the point.
# app.py — sample solution for the packet. Proposed example, not a benchmark.
import json
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path
from urllib.parse import urlparse
ROOT = Path(__file__).resolve().parent
DATA_DIR = ROOT / "data"
DATA_FILE = DATA_DIR / "intents.jsonl"
def load_records():
records = {}
if not DATA_FILE.exists():
return records
with DATA_FILE.open(encoding="utf-8") as fh:
for line in fh:
line = line.strip()
if not line:
continue
row = json.loads(line)
records[row["id"]] = row
return records
class Handler(BaseHTTPRequestHandler):
def log_message(self, fmt, *args):
return
def _send(self, code, payload):
body = json.dumps(payload).encode("utf-8")
self.send_response(code)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def do_GET(self):
if urlparse(self.path).path == "/healthz":
self._send(200, {"ok": True})
return
self._send(404, {"error": "not_found"})
def do_POST(self):
if urlparse(self.path).path != "/v1/intents":
self._send(404, {"error": "not_found"})
return
key = self.headers.get("Idempotency-Key")
if not key:
self._send(400, {"error": "missing_idempotency_key"})
return
length = int(self.headers.get("Content-Length", "0") or 0)
raw = self.rfile.read(length)
try:
body = json.loads(raw.decode("utf-8"))
amount = body["amount_cents"]
currency = body["currency"]
if type(amount) is not int or amount <= 0 or currency != "USD":
raise ValueError("schema")
if set(body.keys()) != {"amount_cents", "currency"}:
raise ValueError("schema")
except Exception:
self._send(400, {"error": "invalid_body"})
return
DATA_DIR.mkdir(exist_ok=True)
records = load_records()
if key in records:
prior = records[key]
same = prior["amount_cents"] == amount and prior["currency"] == currency
if same:
self._send(200, prior)
else:
self._send(409, {"error": "idempotency_conflict"})
return
row = {"id": key, "amount_cents": amount, "currency": currency}
with DATA_FILE.open("a", encoding="utf-8") as fh:
fh.write(json.dumps(row) + "\n")
self._send(201, row)
if __name__ == "__main__":
server = ThreadingHTTPServer(("127.0.0.1", 8088), Handler)
server.serve_forever()
Notice what is missing. There is no config loader. There is no structured logger. There is no repository interface “for later.” Later is how technical debt arrives when generation is cheap. You asked for a door latch. The sample solution is a door latch.
How models and candidates actually fail it
The first failure mode is the phantom cache. The author stores keys in Redis, or in an in-memory dict only, and then adds a comment that “you can swap in a file later.” The tests may still pass if the process never restarts. Your packet should, on a second pass, kill the subprocess and boot it again. Persistence that lives in RAM is a costume.
The second failure mode is schema generosity. Extra JSON fields get silently stored. amount_cents arrives as a float and is rounded. currency is lowercased. That feels polite. It is also a contract change you did not sign. Idempotency bugs love polite parsers.
The third failure mode is the helpful platform. Docker Compose, .env.example, Prometheus, OpenAPI, a workers/ package, and a markdown architecture decision record for a service with two routes. None of that is evil in a real company. All of it is a tell in this packet. The author optimized for looking prepared instead of matching the prompt.
The fourth failure mode is conflict amnesia. Duplicate keys with a different body return 200 and the new body, or 201 and a second line. That is the payments bug you are pretending to screen for. If you only assert happy-path replay, you will hire it.
The fifth failure mode is test capture. The model rewrites test_intents.py so the suite agrees with the extra stack. Score that as a zero on constraint fidelity even if the demo is pretty. Changing the exam is not passing the exam.
Limitations, and who should skip this
This packet does not measure system design. If the role is “build a globally replicated ledger,” you would be foolish to reject Redis on purpose. It also does not measure taste under ambiguous product requirements. The requirements here are hostile and small. That is a filter, not a portrait.
It will not tell you whether a free model is “good.” It will tell you whether this model, today, on this prompt, can leave a hole unfilled. Models drift. Servers get replaced. A single green run is a snapshot, not a vendor evaluation. Do not turn the rubric into a leaderboard and do not cite it as a benchmark. You did not collect one.
Skip the packet if you cannot read the diff. Skip it if your team wants the model to propose architecture. Skip it if the take-home would be given to a junior candidate as their only signal. A closed spec punishes exploration, and some humans deserve a wider prompt.
Use it when the risk in front of you is silent scope. That risk shows up in agent workflows, in AI-authored PRs, and in take-homes that come back over-complete. The fix is not a longer pep talk. The fix is a prompt that makes extra infrastructure a failing test.
If you want a clean machine to run the unittest against a free model without mixing it into your laptop’s current virtualenv, MonkeyCode’s free server option is a reasonable scratch space. Keep the packet. Change the model. Read the tree before you read the README. The README is where invented Redis goes to look official.
Top comments (0)