You open the candidate zip at 10:02. There is no LRU cache. There is no URL shortener. There is a prompt, a rubric, a sample solution that is allowed to look like a first draft, and a short file named failure modes. That is the interview.
A coding agent can paint a pull request green. You have watched it happen on a quiet laptop with a paid seat. The hire you actually need is the person who can say what green was supposed to mean when the model is whoever is free today and the box is not theirs.
If your take-home still asks for a working function, you are grading autocomplete. Ask for the four files instead. Then run them on one shared endpoint so you are not grading wallets.
The packet, not the demo
Picture a hallway conversation after an onsite. Someone says the agent nailed the refactor. Someone else says it burned the shared budget and labeled every timeout as INFO. Both can be true. The missing object is the contract: what the agent was allowed to send, what it was forbidden to invent, and how you would know it cheated the meter.
You are not hiring a magician. You are hiring the person who can write that contract before the model starts typing.
The four files are the whole take-home. A prompt the agent will actually see. A rubric a human can score in fifteen minutes. A sample solution that is allowed to be ugly. A failure catalog that names the ways this task dies in the wild. If a candidate cannot produce those four, a green PR is a costume.
This is a worked example, not a war story from a specific hiring loop. Run it before you ship it. The inner bug is small on purpose. The outer task is whether they can specify the work.
Pin the box before you pin the grade
You cannot compare two packets if one candidate ran a frontier model on a personal GPU and the other pasted into a free chatbot on a phone. That is not an interview. That is a hardware raffle.
Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option are useful here for one boring reason: every candidate hits the same remote endpoint and the same machine. You still own the prompt, the tests, and the rubric. The product is the pin, not the punchline.
Treat the shared box like CI. Nobody gets to swap the runner because their laptop is nicer. Point LLM_ENDPOINT at the free server, keep API keys out of the zip, and grade the files they wrote.
File one: the prompt the agent will see
Here is a candidate-facing prompt you can drop in PROMPT.md. It is intentionally bounded. An unbounded prompt is how a take-home turns into a weekend.
You are fixing classify_logs.py on the shared server.
Goal: label each log line ERROR, WARN, or INFO by calling
$LLM_ENDPOINT (OpenAI-style /v1/chat/completions).
Hard limits:
- Do not send the whole file on every line.
- Identical lines must not pay for a second model call.
- If the endpoint is down, fail the run. Do not default to INFO.
- No extra network. No extra files outside /work.
- Stop after 8 model calls that do not change a test.
Deliver: a patch, pytest passing, and a 12-line note on what you refused to send.
Read it out loud. If you cannot hear the stop condition, the agent will keep looping until the free server sighs. The prompt is part of the grade. A novel is a fail.
File two: the rubric you will actually use
Keep the rubric short enough to print. Long rubrics become fan fiction. This one fits on a card.
PROMPT SHAPE (0-3)
3 bounded goal, hard limits, stop rule
1 vibes and "just make it work"
TOKEN SHAPE (0-3)
3 per-call payload is a line (or a tiny window), not the file
1 whole log hitchhikes on every request
FAILURE (0-2)
2 endpoint down => non-zero exit, no fake INFO
0 swallows the error
SAMPLE (0-2)
2 tests would catch the sample if you broke it
0 sample is a screenshot of a chat
FAILURE CATALOG (0-2)
2 names two real death modes with a signal you can grep
0 "it might be slow"
Hire bar: 9/12. A 3 on token shape with a 0 on failure is a no.
Notice what is missing. There is no extra credit for a named model. There is no bonus for a cinematic README. You are scoring whether they can keep a free shared endpoint alive.
File three: the sample, and the bug it has to kill
The inner program is a log classifier that looks harmless. It is not. It sends the entire file with every line, so fifty lines means fifty copies of the same novel. On a free model that is how you turn a take-home into a brownout.
# classify_logs.py — broken on purpose. Example only; run it yourself.
from __future__ import annotations
import json
import os
import sys
import urllib.error
import urllib.request
ENDPOINT = os.environ.get("LLM_ENDPOINT", "http://127.0.0.1:8080/v1/chat/completions")
def classify_line(line: str, whole_file: str) -> str:
payload = {
"messages": [
{"role": "system", "content": "Label the line ERROR, WARN, or INFO. One word."},
{"role": "user", "content": whole_file + "\n\nNow label this line:\n" + line},
]
}
req = urllib.request.Request(
ENDPOINT,
data=json.dumps(payload).encode("utf-8"),
headers={"Content-Type": "application/json"},
method="POST",
)
with urllib.request.urlopen(req, timeout=20) as resp:
body = json.loads(resp.read().decode("utf-8"))
return body["choices"][0]["message"]["content"].strip()
def classify_file(path: str) -> list[str]:
text = open(path, encoding="utf-8").read()
return [classify_line(line, text) for line in text.splitlines() if line]
if __name__ == "__main__":
print("\n".join(classify_file(sys.argv[1])))
A candidate who writes a sample that still concatenates whole_file has not seen the problem. They have seen a chat window. Your sample has to be worse than that, and then slightly better.
A fair sample solution does three dull things. It sends the line, maybe a three-line window. It caches identical lines. It refuses to invent INFO when the socket dies. Regex short-circuit is allowed. Cleverness is not required.
# classify_logs_sample.py — example fix, not a library.
from __future__ import annotations
import json
import os
import re
import urllib.error
import urllib.request
ENDPOINT = os.environ.get("LLM_ENDPOINT", "http://127.0.0.1:8080/v1/chat/completions")
LOCAL = re.compile(r"\b(ERROR|WARN|INFO)\b")
def classify_line(line: str, cache: dict[str, str]) -> str:
if line in cache:
return cache[line]
m = LOCAL.search(line)
if m:
cache[line] = m.group(1)
return cache[line]
payload = {
"messages": [
{"role": "system", "content": "Reply ERROR, WARN, or INFO. One word."},
{"role": "user", "content": line[:500]},
]
}
req = urllib.request.Request(
ENDPOINT,
data=json.dumps(payload).encode("utf-8"),
headers={"Content-Type": "application/json"},
method="POST",
)
try:
with urllib.request.urlopen(req, timeout=20) as resp:
body = json.loads(resp.read().decode("utf-8"))
except urllib.error.URLError as exc:
raise SystemExit(f"endpoint down: {exc}") from exc
label = body["choices"][0]["message"]["content"].strip().split()[0]
if label not in {"ERROR", "WARN", "INFO"}:
raise SystemExit(f"bad label: {label!r}")
cache[line] = label
return label
The sample is a teaching tool. If their sample is longer than the prompt, they are hiding. If their sample still needs a named paid model to make sense, they failed the environment.
The test that makes the rubric real
A rubric without a meter is a speech. This test does not care what the model said. It cares how fat each request was. Fifty lines should not ship fifty copies of the novel.
# test_token_shape.py — run with: python -m pytest test_token_shape.py -q
import json
import classify_logs as cl
class FakeHTTP:
def __init__(self):
self.bytes_out = 0
self.calls = 0
def __call__(self, req, timeout=20):
self.calls += 1
self.bytes_out += len(req.data or b"")
class Resp:
def read(self_inner):
return json.dumps(
{"choices": [{"message": {"content": "INFO"}}]}
).encode()
def __enter__(self_inner): return self_inner
def __exit__(self_inner, *a): pass
return Resp()
def test_payload_stays_near_the_line(monkeypatch, tmp_path):
log = tmp_path / "app.log"
log.write_text("\n".join(f"INFO boot {i}" for i in range(50)) + "\n")
fake = FakeHTTP()
monkeypatch.setattr(cl.urllib.request, "urlopen", fake)
cl.classify_file(str(log))
avg = fake.bytes_out / max(fake.calls, 1)
assert avg < 800, f"average payload {avg:.0f}b; the whole file is hitching a ride"
def test_down_endpoint_is_not_info(monkeypatch, tmp_path):
def boom(req, timeout=20):
raise cl.urllib.error.URLError("refused")
monkeypatch.setattr(cl.urllib.request, "urlopen", boom)
log = tmp_path / "app.log"
log.write_text("kernel panic\n")
try:
cl.classify_file(str(log))
except SystemExit:
return
raise AssertionError("down endpoint must not become a fake INFO")
Run it locally first, then on the shared server with the free model endpoint filled in. If pytest is green on a laptop and red on the pin, believe the pin. That gap is the interview.
python -m pytest test_token_shape.py -q
LLM_ENDPOINT=http://127.0.0.1:8080/v1/chat/completions \
python classify_logs.py /tmp/app.log; echo exit:$?
You are not measuring tokens with a vendor dashboard. You are measuring bytes on the wire. Bytes do not care about the marketing name of the model.
File four: how this task actually dies
Failure modes are not a vibe. They are scenes with a grep.
One death is the novel-in-the-payload. The signal is average request size climbing with file length. If doubling the log doubles the payload, the agent did not read the prompt. It glued.
Another death is courtesy INFO. The endpoint is down, the function returns INFO, and a pager stays quiet. The signal is a zero exit after a refused connection. You already have a test for that. If their catalog does not mention it, they wrote a happy path.
A third death is cache blindness. The log is 4,000 copies of the same line. The agent still makes 4,000 calls. The free server is not angry at the candidate. It is gone.
A fourth death is the prompt with no stop. The agent retries, reformats, retries, writes a new helper, retries. Eight useless calls was the rail. Without it you are grading stamina.
Candidates fail this take-home in patterned ways. They ship a rubric that only says tests pass. They paste a two-thousand-word prompt that never names the endpoint. Their sample still sends whole_file. They assume a specific paid model and leave LLM_ENDPOINT unset. They never run on the shared box, so they never see the timeout that their laptop hid.
When you score, read the failure catalog before the patch. People reveal what they fear. If they only fear slow CI, they have not met a shared free model yet.
What you do in the debrief
Do not watch them drive the agent live. That is theatre. Sit with the four files. Ask why the prompt forbids extra network. Ask which test would still pass if the sample started concatenating the file again. If they cannot point at avg < 800, they cannot defend the rubric.
Then ask what they would cut if the free server got noisier tomorrow. The right answer is payload and retries, not a shopping list of bigger models. Constraint is the point. Anyone can look smart with an unlimited meter.
Limitations, and who should not use this
This packet is for roles that will specify agent work: platform, SRE-adjacent, eval, internal tools. It is a poor filter for a junior who has never seen an HTTP client. It is a worse filter for a staff engineer whose job is systems design and who will not touch a model that week.
Do not use it as a secret IQ test. The inner bug is obvious once you look at whole_file. The outer skill is writing the rails. If you hide the rails, you are grading luck.
A free shared server is noisy. Neighbors exist. Timeouts exist. That is a feature for this interview and a problem for a production classifier. Do not promote the sample to a service. Do not claim a latency number you did not measure. Do not bake a model name into the rubric; names move and the pin should not.
If your company already standardizes on one paid seat for every candidate, you may not need a free endpoint. You still need the four files. The zip is the method. The free model and free server are only how you stop the raffle.
Ship the zip with a dead classifier, a live test, and a rubric that fits on a card. If you want every candidate on the same free model and the same free server while they write it, MonkeyCode can be that pin. Keep the rest in git. Hire the person whose failure modes you could grep.
Top comments (0)