Grade the Loop Budget Before You Grade the PR

You open the take-home zip at 9:41 p.m. The README is cheerful. The candidate says an agent “iterated until green.” You run make test and the laptop fan climbs. Twelve minutes later the helper is still retrying a 404 against a URL that never existed. The feature is a to-do API. The real defect is that nobody told the loop when to die.

That scene is not rare right now. Feeds are full of agents, tool glue, and claims that models already write the pull request. What those posts skip is the interview problem sitting in your inbox. If you let a candidate — or their agent — touch a free model and a free scratch machine, you are not grading syntax. You are grading whether they can put a budget on a process that wants to run forever.

Free compute is a trap with a friendly face. It removes the credit-card scare that used to stop a runaway retry. It does not remove heat, log spam, or the interviewer who has to replay the session on Monday. So stop asking for a pretty feature list. Ask for a spend contract, then grade that contract the way you would grade a circuit breaker.

The packet you actually send

Keep the product tiny so the loop has nowhere to hide. You want a service so small that extra tool calls look like panic, not architecture. Four hours on the clock. One README. One command that must exit.

Here is the prompt as you would paste it. Label it as a proposed packet if your hiring team has not run it yet.

# Take-home: bounded agent against a sticky 404

You get a toy HTTP service with three routes:

- GET  /health -> 200 {"ok": true}
- GET  /items  -> 200 {"items": []}
- POST /items  -> 201 {"id": "1"} on valid JSON
- GET  /secret -> 404 always. There is no secret. Do not invent one.

You may use a coding model and a scratch server. You may not point either at our
prod credentials, customer data, or this take-home's private test keys.

Deliverables (all four, or the packet is incomplete):

1. `agent_run.py` — a driver that may call a model, may call tools, must stop.
2. `ledger.jsonl` — one line per model or tool call: ts, kind, name, bytes_in,
   bytes_out, error, duplicate_of.
3. `STOP.md` — the stop rule in plain English, including what you do on a
   repeated error.
4. `replay.sh` — interviewer runs this with no chat history. It must exit 0
   or 1 in under 60 seconds on a cold machine.

Hard limits, baked into your driver, not into a promise:

- max 12 model calls
- max 20 tool calls
- max 3 identical errors in a row, then halt
- wall clock 45 seconds for `replay.sh`
- if /secret returns 404, you record it once and you do not retry it

We grade the ledger and the stop. We do not grade how clever the model sounded.

Notice what you refused to ask for. No design doc about “multi-agent orchestration.” No extra infrastructure. The dead-end route is the point, and the candidate already knows it is a 404. If their loop still knocks on that door, the packet did its job.

You do not need a production cluster to practice this. A free model endpoint and a free scratch server are enough. If you already have MonkeyCode around, its free model access and free server option fit the exercise without turning the take-home into a procurement meeting. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Those two availability claims are the only product facts used here. No model names, quotas, or hardware stories.

What “good” looks like in code

Talk is cheap, so put the budget in a type the interviewer can import. The sample below is a proposed driver, not a claim that a particular candidate shipped it.

# loop_budget.py
from __future__ import annotations

import json
import time
from collections import deque
from dataclasses import dataclass, asdict
from pathlib import Path
from typing import Callable, Deque, Literal

Kind = Literal["model", "tool"]


@dataclass
class Event:
    ts: float
    kind: Kind
    name: str
    bytes_in: int
    bytes_out: int
    error: str | None
    duplicate_of: int | None


class BudgetBlown(RuntimeError):
    pass


class LoopBudget:
    def __init__(
        self,
        ledger_path: Path,
        max_model: int = 12,
        max_tool: int = 20,
        max_identical: int = 3,
        wall_s: float = 45.0,
    ) -> None:
        self.ledger_path = ledger_path
        self.max_model = max_model
        self.max_tool = max_tool
        self.max_identical = max_identical
        self.deadline = time.monotonic() + wall_s
        self.events: list[Event] = []
        self._errors: Deque[str] = deque(maxlen=max_identical)
        self.ledger_path.write_text("")

    def _check_clock(self) -> None:
        if time.monotonic() > self.deadline:
            raise BudgetBlown("wall clock")

    def record(self, kind: Kind, name: str, bytes_in: int, bytes_out: int,
               error: str | None) -> Event:
        self._check_clock()
        model_n = sum(1 for e in self.events if e.kind == "model")
        tool_n = sum(1 for e in self.events if e.kind == "tool")
        if kind == "model" and model_n >= self.max_model:
            raise BudgetBlown("model calls")
        if kind == "tool" and tool_n >= self.max_tool:
            raise BudgetBlown("tool calls")

        dup = None
        if error:
            self._errors.append(error)
            if len(self._errors) == self.max_identical and len(set(self._errors)) == 1:
                raise BudgetBlown(f"repeated error: {error}")
            for i, prev in enumerate(self.events):
                if prev.error == error and prev.name == name:
                    dup = i
                    break
        else:
            self._errors.clear()

        ev = Event(time.time(), kind, name, bytes_in, bytes_out, error, dup)
        self.events.append(ev)
        with self.ledger_path.open("a") as fh:
            fh.write(json.dumps(asdict(ev)) + "\n")
        return ev

    def call_tool(self, name: str, fn: Callable[[], tuple[int, int, str | None]]):
        """fn returns (bytes_in, bytes_out, error)."""
        bytes_in, bytes_out, error = fn()
        return self.record("tool", name, bytes_in, bytes_out, error)

The interesting line is duplicate_of. A free model will happily re-explain a 404 in warmer language. Your ledger should not. Once /secret fails, the next identical failure is a budget event, not a plot twist. That is the whole analogy: treat the agent like a guest with a meal card, not like a roommate with your pantry.

Wire a sticky 404 so the candidate cannot “fix the server” and dodge the stop rule.

# sticky_404.py
import json
from http.server import BaseHTTPRequestHandler, HTTPServer

class Handler(BaseHTTPRequestHandler):
    def _send(self, code: int, payload: dict) -> None:
        body = json.dumps(payload).encode()
        self.send_response(code)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def do_GET(self) -> None:
        if self.path == "/health":
            return self._send(200, {"ok": True})
        if self.path == "/items":
            return self._send(200, {"items": []})
        if self.path == "/secret":
            return self._send(404, {"error": "no such route"})
        return self._send(404, {"error": "missing"})

    def do_POST(self) -> None:
        if self.path != "/items":
            return self._send(404, {"error": "missing"})
        n = int(self.headers.get("Content-Length", "0"))
        raw = self.rfile.read(n)
        try:
            json.loads(raw or b"{}")
        except json.JSONDecodeError:
            return self._send(400, {"error": "bad json"})
        return self._send(201, {"id": "1"})

    def log_message(self, fmt: str, *args) -> None:
        return

if __name__ == "__main__":
    HTTPServer(("127.0.0.1", 8077), Handler).serve_forever()

replay.sh should be boring. Interviewers are tired. Boring is a feature.

#!/usr/bin/env bash
set -euo pipefail
python sticky_404.py & srv=$!
cleanup() { kill "$srv" 2>/dev/null || true; }
trap cleanup EXIT
sleep 0.3
python agent_run.py --base http://127.0.0.1:8077 --ledger ledger.jsonl
test -s ledger.jsonl
python - <<'PY'
import json, pathlib, sys
rows = [json.loads(l) for l in pathlib.Path("ledger.jsonl").read_text().splitlines() if l]
secrets = [r for r in rows if r.get("name") == "GET /secret"]
if len(secrets) > 1:
    sys.exit("retried the documented 404")
if len(rows) > 32:
    sys.exit("ledger longer than the published budget")
print(f"ok {len(rows)} events")
PY

Run the grader the way you will on Monday, not the way a demo video runs.

chmod +x replay.sh
python -m pytest test_loop_budget.py -q
./replay.sh

The rubric lives in a test, not in a vibe

You already know what happens if the score lives in a spreadsheet named feelings.xlsx. Two interviewers will argue about “initiative.” Put the score next to the ledger instead. The test file is the rubric. If a human still wants a narrative, they can read STOP.md after the tests pass.

# test_loop_budget.py
from pathlib import Path
import pytest
from loop_budget import BudgetBlown, LoopBudget

def test_repeated_404_trips_the_stop(tmp_path: Path):
    b = LoopBudget(tmp_path / "ledger.jsonl", max_identical=3, wall_s=5)
    def boom():
        return 24, 32, "GET /secret -> 404"
    b.call_tool("GET /secret", boom)
    b.call_tool("GET /secret", boom)
    with pytest.raises(BudgetBlown, match="repeated error"):
        b.call_tool("GET /secret", boom)

def test_model_cap(tmp_path: Path):
    b = LoopBudget(tmp_path / "ledger.jsonl", max_model=2, wall_s=5)
    b.record("model", "plan", 10, 40, None)
    b.record("model", "plan", 10, 40, None)
    with pytest.raises(BudgetBlown, match="model calls"):
        b.record("model", "plan", 10, 40, None)

def test_clock(tmp_path: Path, monkeypatch):
    b = LoopBudget(tmp_path / "ledger.jsonl", wall_s=0.01)
    monkeypatch.setattr("loop_budget.time.monotonic", lambda: b.deadline + 1)
    with pytest.raises(BudgetBlown, match="wall clock"):
        b.record("tool", "noop", 0, 0, None)

Pass those three and the candidate has earned a conversation. Fail the repeated-404 test and you can end the loop — the interview loop — without a debate about formatting. Style is downstream. The stop button is the feature.

Failure modes you will see by Wednesday

The first packet looks confident and empty. STOP.md says “we halt on errors,” but agent_run.py catches Exception and continues. The ledger is a single line that reads ran ok. That is not a ledger. That is a press release. You fail it for missing evidence, not for missing poetry.

The second packet retries /secret because the model “wanted to be sure.” Certainty is not a budget. The duplicate column will light up like a smoke alarm if they recorded honestly. If they did not record the retries, you fail them for a forged ledger. Either way the take-home worked. You did not hire a retry storm with a smile.

The third packet spends the whole budget planning. Twelve model calls, zero tool calls, a beautiful essay about REST. Free model access makes that essay feel free. It is not free for you, because you still have to read it. replay.sh should exit non-zero when the ledger never touched GET /items. An agent that only talks is a blog post in a trench coat.

The fourth packet is sneakier. It shells out to a long-running helper on the scratch server and background-jobs the work so replay.sh can exit in two seconds. Your wall-clock cap on the driver does not see the orphan. Add one line to the rubric in STOP.md: no leftover processes, and replay.sh must pgrep -f agent itself after exit and find nothing. Free servers still have PIDs. PIDs still lie if you do not look.

What this packet is not for

Do not use this on a intern screen that is supposed to test hash maps. You will punish people for not play-acting an agent. Do not use it if your legal team has not approved sending take-home text to a third-party model. The candidate’s NDA is not a hallucination you can retry away. Do not use it as a stealth load test against someone else’s endpoint. The sticky 404 lives on localhost for a reason.

The method also fails if you inflate the product. The moment you ask for Kubernetes, a queue, and a dashboard, the candidate can hide a runaway loop behind “eventual consistency.” Keep the service boring so the budget is loud.

Limitations travel with the sample code. LoopBudget does not know about tokens, GPUs, or rate-limit headers. It counts calls and identical error strings. Two different 404 bodies will look like progress. If you need tighter matching, hash the route plus the status code, not the model’s apology. The wall clock uses monotonic on one machine. It will not save you from a candidate who runs the heavy work elsewhere and only ships the receipt.

Close the loop on your side too

After you collect a few packets, read the ledgers before the READMEs. You will feel the difference in your shoulders. The good ones are short. They hit /health, create an item, record the forbidden 404 once, and stop. The bad ones narrate. Narration is a smell when the budget is twelve.

If you want a place to rehearse the packet before you send it to a human, a free model plus a free server is the whole stage. MonkeyCode is one option for that rehearsal. Then go grade the stop condition, and let the PR wait its turn.