The Queue Is the Curriculum: Grading a Bootcamp Lab That Shares One Free Tier

Two students ship the same agent. Same repo, same prompt, same free tier. One gets a reply in 900 ms; the other waits eleven seconds behind a queue they cannot see.

Who wrote the better agent?

Here's the conclusion I'd stake the lab on: when the tier is shared, you grade the distribution, not the demo. And you calibrate the rubric before you grade a single student, or you are just ranking luck.

This is the lab I now hand to a cohort: a jittered load harness, four checkpoints, a rubric that lives in a file, and a kappa gate between two graders.

Why a shared free tier breaks your grading

My first version of this lab graded p50 latency. It was a disaster in exactly the way you'd predict.

• The student who ran at 07:00 got a quiet tier and a gold star.
• The student who ran at 20:00 got a noisy one and a rewrite request.
• Both had shipped identical code.

Absolute numbers on a shared tier measure the neighbourhood, not the submission. So the lab now grades shape: error rate, the ratio between tail and median, whether limits are declared, and whether a stranger can reproduce your verdict.

Setup: three files and a Makefile

lab/
  harness/contend.py     # jittered wave, stdlib only
  rubric/rubric.py       # weights + calibration
  reports/               # wave-*.json, committed
  Makefile

check:                      ## 5 seats, 20 calls, must be green twice
    python harness/contend.py --url $${LAB_URL} --seats 5 --n 20 --spread 5

wave:                       ## the graded run
    python harness/contend.py --url $${LAB_URL} --seats 30 --n 120 --spread 20 --budget 8

Students point LAB_URL at their own endpoint. For the model calls behind it, the cohort uses MonkeyCode's free model access, and the endpoint itself runs on its free server option, so nobody starts the week with a billing conversation.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

One caution before you plan a semester around any free offer: quotas and capacity terms change. The allowance I was briefed on was 10 million tokens and a free server — treat that as a number to re-verify against the current terms each cohort, not as a constant in your syllabus.

The harness: no deps, no excuses

# harness/contend.py — Python 3.10+, standard library only
import argparse, asyncio, json, os, random, time, urllib.error, urllib.request
from pathlib import Path

def pct(xs, p):
    if not xs:
        return None
    xs = sorted(xs)
    return xs[round((p / 100) * (len(xs) - 1))]

async def call(url, payload, timeout=120):
    t0 = time.perf_counter()
    def _do():
        req = urllib.request.Request(
            url, data=json.dumps(payload).encode(),
            headers={"content-type": "application/json"})
        with urllib.request.urlopen(req, timeout=timeout) as r:
            return r.status
    try:
        status = await asyncio.to_thread(_do)
        return {"ok": 200 <= status < 300, "status": status,
                "ms": (time.perf_counter() - t0) * 1000}
    except Exception as e:  # lab code: log the class, keep the wave going
        return {"ok": False, "status": None, "err": type(e).__name__,
                "ms": (time.perf_counter() - t0) * 1000}

async def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--url", required=True)
    ap.add_argument("--seats", type=int, default=30)
    ap.add_argument("--n", type=int, default=120)
    ap.add_argument("--spread", type=float, default=20.0)
    ap.add_argument("--budget", type=int, default=8)   # max in flight
    a = ap.parse_args()

    sem = asyncio.Semaphore(a.budget)

    async def seat_call(i):
        await asyncio.sleep(random.uniform(0, a.spread))
        async with sem:
            return await call(a.url, {"seat": i % a.seats, "probe": i})

    t0 = time.perf_counter()
    rs = await asyncio.gather(*(seat_call(i) for i in range(a.n)))
    wall = time.perf_counter() - t0

    ms = [r["ms"] for r in rs if r["ok"]]
    errs = [r for r in rs if not r["ok"]]
    rep = {
        "seats": a.seats, "n": a.n, "spread_s": a.spread, "budget": a.budget,
        "wall_s": round(wall, 2), "ok": len(ms), "errors": len(errs),
        "error_rate": round(len(errs) / a.n, 3),
        "p50_ms": round(pct(ms, 50), 1) if ms else None,
        "p95_ms": round(pct(ms, 95), 1) if ms else None,
        "tail_ratio": round(pct(ms, 95) / pct(ms, 50), 2) if ms else None,
        "err_kinds": sorted({r.get("err") for r in errs if r.get("err")}),
        "git_sha": os.popen("git rev-parse --short HEAD").read().strip(),
    }
    Path("reports").mkdir(exist_ok=True)
    out = Path("reports") / f"wave-{int(time.time())}.json"
    out.write_text(json.dumps(rep, indent=2))
    print(json.dumps(rep, indent=2))

if __name__ == "__main__":
    asyncio.run(main())

Why --budget? Because an unbounded wave measures your socket table, not the server. Eight in flight is the number I use for a 30-seat cohort; pick your own and write it down, because it is part of the experiment.

Checkpoints

Each one has a pass condition a grader can check without reading your code.

1. C1 — single cold call. One request, one logged transcript. Pass: the JSON body and the log line are committed.
2. C2 — the double run. make check green twice in a row, same params, same verdict. Pass: two reports/wave-*.json with matching p50 shape and zero errors.
3. C3 — the cohort wave. 120 calls, 30 seats, 20 s jitter. Pass: error_rate == 0 and a declared failure story for when it isn't.
4. C4 — the break point. Rerun with --budget 32 and --spread 0. Pass: the report shows where it falls over, not that it survived.

C4 is the one students hate and the one that teaches the most. "It held" is not a result. "It held to 16 in flight, then the tail ratio crossed 4×" is a result.

The rubric is a file with weights

# rubric/rubric.py
CRITERIA = {
    "correctness":  (0.30, "wave json: ok > 0, every error has a named class"),
    "bounded_load": (0.25, "tail_ratio under the cohort ceiling you declared"),
    "reproducible": (0.25, "two runs, fresh clone, same verdict"),
    "honest_limits":(0.20, "README says where the lab breaks and why"),
}

def score(evidence: dict) -> float:
    total = 0.0
    for name, (weight, _) in CRITERIA.items():
        total += weight * max(0.0, min(1.0, evidence.get(name, 0.0)))
    return round(total * 100, 1)

Criterion	Weight	Evidence a grader looks at	Auto-fail
Correctness	30	reports/*.json, ok > 0, named error classes	100% errors with no explanation
Bounded load	25	tail_ratio vs. declared ceiling	ceiling declared after the run
Reproducible	25	second run from a fresh clone	results unreproducible and unmentioned
Honest limits	20	README "where this breaks" section	claims of unlimited capacity

Notice there is no line for speed. Speed is not a criterion when the tier is shared. The ceiling isn't a fixed millisecond value either; declare it in week one from three baseline runs and freeze it.

Calibrate the graders before you grade the cohort

This part is non-negotiable, and it's the thing almost nobody does. Two graders score the same five anchor submissions independently.

def kappa(a, b, k=4):
    """a, b: integer scores 0..k-1 from two graders. Report agreement honestly."""
    n = len(a)
    po = sum(1 for x, y in zip(a, b) if x == y) / n
    pe = sum((a.count(i) / n) * (b.count(i) / n) for i in range(k))
    return 1.0 if pe == 1 else (po - pe) / (1 - pe)

• kappa ≥ 0.6: publish the rubric and start grading.
• kappa < 0.6: the rubric is broken. Rewrite the anchor examples. Do not blame the student, and do not average the two scores to make the number look tidy.

A rubric nobody calibrated is a vibe with a spreadsheet attached. Sound familiar?

Stretch goals

• Backpressure: return 429 with Retry-After when in-flight exceeds budget, and prove the harness respects it.
• Idempotency keys: retry the wave twice with the same probe ids and show you did not double-charge your own token budget.
• Per-seat ledger: log tokens per seat and plot spend against probe index.
• Swap the grader: have a classmate run the rubric against your report and see if they land on the same score.

Limitations, and who should skip this

Be honest with yourself about what this lab can and cannot prove.

• No SLA. A free server option is shared capacity. Your p95 is a snapshot of one afternoon, and a noisy neighbour can invalidate it.
• No permanence guarantee. Free tiers change. Re-verify the terms and the token allowance before you build them into a syllabus.
• Not a benchmark of the product. This harness measures your endpoint's behaviour under contention. It says nothing about model quality.
• Don't use this if you need guaranteed throughput, contractual uptime, data isolation, or reproducible latency numbers for a published comparison. Pay for capacity, or measure on hardware you control.

Try the smallest version first

Don't invite 30 students into a lab you haven't run alone. Spin one endpoint, fire five seats and twenty calls, and look at the JSON. If tail_ratio is already ugly at that size, your cohort will spend day three debugging your harness instead of their agent — and that's my fault, not theirs.

Start with make check. The queue will tell you the rest.