Grade the Deploy, Not the Repo

You have ninety minutes and six take-homes waiting. Five of them run perfectly in the candidate's README GIF. One of them runs on a free tier, and the first request after twenty minutes of idle takes eleven seconds. Which repo actually told you something?

Code review grades what was written. Almost nobody grades what happens when the thing is asked to work after it has been asleep, which is the state most small services actually live in. So here is a task, a rubric, a sample solution, and the failure modes you will keep seeing.

The prompt you hand over

Give a narrow spec and a hard number. Candidates do better when the constraint is numeric and everything else is theirs.

Build one endpoint and deploy it to a free-tier host.

POST /summarize  {"text": "..."} -> {"summary": "...", "model_ms": int}

1. Backed by a real LLM call. Any provider. Must run without a paid plan.
2. GET /health returns 200 in under 200 ms and never calls the model.
3. Cold start (first request after >= 15 min idle) completes in < 5 s
   end to end, or returns a documented 503 with Retry-After.
4. No secrets in the repo or the image. Config comes from the environment.
5. README states your cold-start number and how you measured it.

Deliver the URL, the repo, and one paragraph on what you would change
with a real budget.

That last line does more work than it looks like. It separates candidates who understood the tradeoff from candidates who only made the test pass.

Why the free tier belongs inside the task

A take-home that needs a credit card filters for credit cards, which is a bad filter and everyone knows it. But the constraint cuts deeper than fairness. If the model call and the host both cost money on day one, a large share of candidates will stub the model with return 'summary' and you will never watch them handle a slow upstream or a 429.

This is the one place where the tooling choice is genuinely load-bearing. MonkeyCode advertises free model access and a free server option, so the LLM call and the deploy target can both sit inside the unpaid path. The operator currently quotes a 10-million-token allowance; treat that as a dated offer, not a constant baked into your rubric. Put the terms in the prompt, link the live page, and re-read them before every hiring round, because free tiers age fast.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two things to write down next to the prompt. Free tiers sleep, throttle, and change shape without warning, which is exactly why requirement 3 exists and why the rubric rewards a documented failure over a mystery timeout. And a candidate who picks a different free host, or a different free model, is not failing the task. You are grading behavior, not a vendor.

The sample solution, and the one seam that matters

Keep exactly one file that knows which provider you used. Everything else talks to that file and nothing else.

# llm.py - the only file that names a provider
import os, time

def summarize(text: str) -> tuple[str, int]:
    start = time.monotonic()
    # Swap this for the SDK call in your provider's docs.
    out = _call_provider(text, api_key=os.environ['LLM_API_KEY'])
    return out, int((time.monotonic() - start) * 1000)

Then the service, with a cheap health check and a deadline around the model call.

# app.py - FastAPI sketch
import asyncio, os, time
from fastapi import FastAPI
from fastapi.responses import JSONResponse
from llm import summarize

app = FastAPI()
BUDGET_S = float(os.environ.get('COLD_BUDGET_S', '5'))
READY = False

@app.on_event('startup')
async def warm():
    global READY
    READY = True  # <- a lie for the first second or two

@app.get('/health')
def health():
    return {'ready': READY}

@app.post('/summarize')
async def do(body: dict):
    t0 = time.monotonic()
    try:
        summary, model_ms = await asyncio.wait_for(
            asyncio.to_thread(summarize, body['text']), timeout=BUDGET_S - 0.5
        )
    except asyncio.TimeoutError:
        return JSONResponse(
            status_code=503,
            content={'error': 'cold_start'},
            headers={'Retry-After': '3'},
        )
    return {'summary': summary, 'model_ms': model_ms,
            'total_ms': int((time.monotonic() - t0) * 1000)}

The comment inside warm() is the point of the exercise. A startup hook fires before the process can serve traffic well, so /health claims ready while the first real request is still paying import costs. I want the candidate to notice that, or at least to explain it when you ask.

Now the harness. Standard library only, so it runs on the reviewer's laptop with nothing installed and nothing to trust.

#!/usr/bin/env python3
# probe.py - usage: python probe.py https://app.example --idle 900 --samples 5
import json, statistics, sys, time, urllib.request, urllib.error

BASE = sys.argv[1].rstrip('/')
IDLE = int(sys.argv[sys.argv.index('--idle') + 1])
SAMPLES = int(sys.argv[sys.argv.index('--samples') + 1])

def hit(path, payload=None, timeout=20):
    body = json.dumps(payload).encode() if payload else None
    req = urllib.request.Request(BASE + path, data=body,
                                 headers={'content-type': 'application/json'})
    t0 = time.monotonic()
    try:
        with urllib.request.urlopen(req, timeout=timeout) as r:
            return r.status, int((time.monotonic() - t0) * 1000)
    except urllib.error.HTTPError as e:
        return e.code, int((time.monotonic() - t0) * 1000)
    except Exception as e:
        return type(e).__name__, int((time.monotonic() - t0) * 1000)

print('health', hit('/health'))
print(f'idling {IDLE}s...')
time.sleep(IDLE)
cold = [hit('/summarize', {'text': 'x' * 200}) for _ in range(SAMPLES)]
warm = [hit('/summarize', {'text': 'x' * 200}) for _ in range(SAMPLES)]
for label, rows in (('cold', cold), ('warm', warm)):
    ok = [ms for code, ms in rows if code == 200]
    print(label, rows)
    if ok:
        print(f'  n={len(ok)} p50={int(statistics.median(ok))}ms max={max(ok)}ms')

Run that against the URL in the submission and paste the output into your review notes. Two status lists and a median beat any amount of prose in a README.

The shape of the output looks like this, and it is illustrative only — your numbers will move with host, region, and time of day:

health (200, 84)
idling 900s...
cold [(503, 4870), (200, 3120), (200, 2410), (200, 2380), (200, 2405)]
  n=4 p50=2405ms max=3120ms
warm [(200, 1210), (200, 1190), (200, 1188), (200, 1205), (200, 1197)]
  n=5 p50=1197ms max=1210ms

That lone 503 followed by clean recoveries is a candidate who read requirement 3 and took it seriously. A run where every cold call returns 200 after nine seconds is a candidate who never measured anything, and the README will usually admit it in the same breath.

Rubric, narrated

Seam, 25 points. One adapter, no provider names leaking into route handlers, every variable read from the environment. Budget honesty, 25. The cold-start claim reproduces under your probe, or the service returns 503 with Retry-After and says why. Health semantics, 15. /health is cheap, truthful, and not a model call wearing a hat. Repo hygiene, 15. No .env anywhere in git log, no key baked into the image, a .env.example listing every variable the app reads. Failure handling, 10. Timeout, 429, and empty input each produce a defined response instead of a stack trace. The closing paragraph, 10. Names a real tradeoff: caching the summary, a scheduled warm ping, a queue, or paying for something reserved.

Here is what those criteria look like in the wild.

Symptom	What it usually means	Question to ask
/health takes 900 ms	health check calls the model	"What does ready mean here?"
Cold calls always 200, always slow	budget never enforced	"Where is the deadline?"
503 with no Retry-After	timeout handled, contract ignored	"What should the client do next?"
.env present in history	secret hygiene skipped	"How would you rotate it?"
README number unreproducible	measured once, or never	"Show me the command."
Retry loop on 429	no backoff, no jitter	"What happens at 3 a.m.?"

Who should not use this

Skip it if your production stack has no cold starts. You would be grading a skill your team never exercises, and the candidate spends a weekend on a problem you do not have. Skip it if you cannot keep a stable link to current free-tier terms, because a task whose constraints expire mid-round is unfair in a way no rubric repairs. Skip it if the role never touches deploy or configuration; a data-structure task grades sharper for that hire. And if you are the candidate reading this: free tiers are cheap, not dependable, and the whole point is to say so in your README before a reviewer discovers it for you.

If you want to feel the constraint before you impose it, run this prompt on yourself once. Free model access and the free server option are the shortest path to a genuine cold start on an ordinary Tuesday night.