The Take-Home With a Token Budget

You rerun a candidate's submission on a Tuesday morning and it dies on request twelve. Not a crash — a 401. The agent worked beautifully for the two days it was recorded, because it worked on a key that expired the moment the invoice did.

That submission was never really about the candidate's code. It was about their credit card. And you spent an hour of reviewer time discovering it.

Deadline is not the only constraint

Most take-homes test one axis: can you build it before Sunday? Almost none test the axis that decides whether the thing survives contact with production, which is can you build it inside a budget you do not control.

So change the prompt. Hand out a token ceiling and a free server, and ask for a ledger. Nothing about the task gets harder to grade — it gets easier, because now the submission has numbers attached to it instead of adjectives.

Two operator-supplied facts make this practical: MonkeyCode currently offers free model access and a free server option, which is enough for a candidate to complete the exercise without a card on file.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The operator also advertises 10 million free tokens as part of the free tier. Treat that as a claim with a date on it, not a law of physics — quotas and server terms change, so your prompt should tell candidates to read the console themselves rather than trust your blog post. That instruction is itself part of the test.

The prompt

Keep it short. Long prompts teach candidates to guess what you want.

Build a small service that answers questions over the attached 40-document corpus. It must run on a free tier only: free model access plus the free server option. No paid keys, no local GPU.

Deliver three things: the service, a ledger.jsonl recording token spend per request, and a one-page note describing exactly what breaks first when the budget runs out and what the user sees at that moment.

Soft ceiling: whatever the console gives you for free. If you hit it, stop and write down where.

Notice what is absent. No accuracy target, no framework requirement, no request-per-second goal. Those invite optimisation theater. The ledger and the breakdown note are the deliverable, because they are the two artifacts a candidate cannot fake in an hour.

The artifact you hand them

Give candidates a harness so the budget is measurable, not vibes. This is illustrative Python, not executed against any specific provider — it wraps whatever HTTP client you point it at.

# budget.py - illustrative harness, provider-agnostic
import json, os, pathlib

LEDGER = pathlib.Path("ledger.jsonl")
CEILING = int(os.environ.get("TOKEN_CEILING", "200000"))

class BudgetExhausted(Exception):
    pass

def spent() -> int:
    if not LEDGER.exists():
        return 0
    rows = (json.loads(line) for line in LEDGER.read_text().splitlines() if line)
    return sum(r["tokens"] for r in rows)

def record(kind: str, tokens: int, ms: int, ok: bool) -> None:
    row = {"kind": kind, "tokens": tokens, "ms": ms, "ok": ok}
    with LEDGER.open("a") as f:
        f.write(json.dumps(row) + "\n")

def call(transport, prompt: str, kind: str = "answer") -> str:
    remaining = CEILING - spent()
    if remaining <= 0:
        raise BudgetExhausted(f"ceiling {CEILING} reached")
    started = os.times().elapsed
    try:
        text, tokens = transport(prompt)
    except Exception:
        # failed calls still cost reviewer time; log them, tokens unknown -> 0
        record(kind + ":error", 0, 0, ok=False)
        raise
    record(kind, tokens, int((os.times().elapsed - started) * 1000), ok=True)
    return text

The second file is the one that separates candidates. A test that proves the ceiling actually holds:

# test_budget.py
import pytest, budget

def fake_ok(prompt):
    return "answer", 120_000

def fake_boom(prompt):
    raise RuntimeError("upstream 503")

def test_ceiling_raises_instead_of_overspending(tmp_path, monkeypatch):
    monkeypatch.setattr(budget, "LEDGER", tmp_path / "ledger.jsonl")
    monkeypatch.setattr(budget, "CEILING", 200_000)
    budget.call(fake_ok, "q1")
    with pytest.raises(budget.BudgetExhausted):
        budget.call(fake_ok, "q2")

def test_failures_are_recorded_not_swallowed(tmp_path, monkeypatch):
    monkeypatch.setattr(budget, "LEDGER", tmp_path / "ledger.jsonl")
    with pytest.raises(RuntimeError):
        budget.call(fake_boom, "q1")
    assert "error" in (tmp_path / "ledger.jsonl").read_text()

Run pytest -q. Two green tests in under a minute tell you more about how someone thinks about money than a five-hundred-line README.

The rubric

Score on judgment, not volume. Four rows, and be explicit that a candidate can pass with an ugly UI.

Criterion	Strong submission	Fails
Ledger honesty	Failed calls and retries appear in the log	Only successful calls counted
Degradation	Budget exhaustion produces a defined user-facing state	Exception traceback, or silent empty answer
Cost accounting	Cost per answered question stated, with the arithmetic shown	"It's cheap"
Cold reality	Notes that a sleeping free server adds wake latency	Claims p95 without measuring a cold start

That last row is where free tiers bite. A free server that idles down means your first request after a quiet period pays a wake tax. A candidate who measures it and says so is telling you they will not be surprised by it in production.

Failure modes you will actually see

The ledger that lies. It counts input tokens but not output, or records only the happy path. Ask one question about the gaps and the whole submission re-reads differently.

The silent fallback. When the model errors, the service returns an empty string with a 200. That is worse than a 500, because monitoring never fires. This is the single most common defect in free-tier deployments, and it is invisible unless your rubric names it.

Demo-path optimisation. Three beautiful scripted queries, no handling of the fourth. The corpus has forty documents; ask for a query that hits the edges.

Cold-start denial. The note claims sub-200ms responses and the deploy config sleeps after inactivity. Not a lie exactly — a measurement gap. Distinguish those two in your feedback.

No exit ramp. The free tier is hardcoded into the architecture: no abstraction between the transport and the rest of the code, so moving to a paid model is a rewrite. Ask directly how they would swap providers in an afternoon.

Limitations, and who should not do this

This format costs reviewer time. Someone has to open the ledger, run the tests, and ask two follow-up questions. If your loop cannot afford thirty focused minutes per candidate, a budget take-home degrades into an unpaid essay contest and you should use a live pairing session instead.

Skip it for roles where data handling dominates — if your legal team cannot tell you whether free-tier inference is acceptable for the synthetic corpus, do not put candidates in that ambiguity. Skip it, too, if you need hard latency guarantees as a hiring signal; a free tier cannot demonstrate an SLA it does not offer.

And be honest about the asymmetry. A budget exercise is still unpaid work. Keep it under two hours, tell candidates the time box, and mean it.

Before you write the prompt

Spend one hour inside the free tier yourself. Hit the limit once, watch what your own code does at request nineteen, and you will design a sharper rubric than any template — including this one.

If you want a zero-cost place to start that hour, the free model access and free server option are enough to run the harness above end to end; give it one run before you hand it to anyone else.