The Budget Is Part of the Answer: A Bootcamp Lab on Grading Agent Runs on a Free Tier

Your agent run isn't finished when the tests go green. It's finished when the tests go green inside the budget you were handed. That's the whole lab.

Most students I've watched treat a free tier as an infinite sandbox. It isn't. A free allowance is a constraint you get for free, which makes it a much better teaching tool than a credit card.

So here's the exercise: one prompt, one free model access tier, one free server box, and a ledger that has to reconcile. The grade is not "did it work." The grade is "can you prove what it cost, and where the cost went."

Why this lab exists (and what it is not)

Three things changed the shape of student agent projects recently. Model output got cheaper and more verbose; runs got longer; and the retry loop became the default debugging strategy. That last one is where budgets die.

This lab is not a benchmark. It's not a comparison of providers. It's a metering discipline you can run on any free tier, including the one this lab assumes.

That assumption: MonkeyCode offers free model access and a free server option, which is what makes a no-money lab possible at all. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I'm not going to restate a quota number here — quota terms shift, and a stale number makes a terrible grading key. The source of truth is the project's own current terms page, which every student screenshots at the start of the lab as their budget contract.

Setup: five steps, twenty minutes

1. Screenshot the terms page. Name it budget_contract.png. This is your ceiling for the whole lab.
2. Pick one task with a hard oracle. Parsing a log format, fixing an off-by-one in a paginator, adding a failing edge case. Something a test can settle yes or no.
3. Refuse to touch the task until logging exists. No logging, no run. That rule saves more time than any prompt trick.
4. Log every call as one JSON line. One line per model call, one line per tool call, never batched.
5. Export the log before anything else. Free servers get recycled. Logs that lived only on the box are logs you don't have.

The event schema is small on purpose:

{"step":"fix_offbyone","attempt":1,"kind":"model_call","prompt_tokens":812,"completion_tokens":204,"reason":"initial"}
{"step":"fix_offbyone","attempt":2,"kind":"model_call","prompt_tokens":1490,"completion_tokens":331,"reason":"test_1_failed"}
{"step":"fix_offbyone","attempt":2,"kind":"tool_call","prompt_tokens":0,"completion_tokens":0,"reason":"pytest"}

Note the reason field. A retry without a reason is, by lab rule, an unlabeled retry, and unlabeled retries cost double in the rubric. You'll see why below.

The artifact: a ledger you can run

This script reduces the log to a per-attempt ledger and prints a retry-overhead ratio. It's deliberately boring code, because boring code is what you can trust at 2 a.m.

# ledger.py -- reduce a JSONL run log into a per-attempt budget ledger.
# Usage: python ledger.py run.jsonl > ledger.csv
import csv, json, sys
from collections import defaultdict

REQUIRED = {"step", "attempt", "kind", "prompt_tokens", "completion_tokens"}
KINDS = {"model_call", "tool_call", "other"}

def load(path):
    rows, bad = [], 0
    with open(path, encoding="utf-8") as fh:
        for lineno, line in enumerate(fh, 1):
            line = line.strip()
            if not line:
                continue
            try:
                ev = json.loads(line)
            except json.JSONDecodeError:
                print(f"skip line {lineno}: not json", file=sys.stderr)
                bad += 1
                continue
            missing = REQUIRED - ev.keys()
            if missing:
                print(f"skip line {lineno}: missing {sorted(missing)}", file=sys.stderr)
                bad += 1
            elif ev["kind"] not in KINDS:
                print(f"skip line {lineno}: unknown kind {ev['kind']!r}", file=sys.stderr)
                bad += 1
            else:
                rows.append(ev)
    return rows, bad

def main():
    rows, bad = load(sys.argv[1])
    agg = defaultdict(lambda: {"calls": 0, "prompt": 0, "completion": 0})
    for ev in rows:
        a = agg[(ev["step"], int(ev["attempt"]))]
        a["calls"] += 1
        a["prompt"] += int(ev["prompt_tokens"])
        a["completion"] += int(ev["completion_tokens"])

    w = csv.writer(sys.stdout)
    w.writerow(["step", "attempt", "calls", "prompt_tokens", "completion_tokens", "total_tokens"])
    totals = {"all": 0, "retry": 0}
    for (step, attempt) in sorted(agg):
        a = agg[(step, attempt)]
        total = a["prompt"] + a["completion"]
        w.writerow([step, attempt, a["calls"], a["prompt"], a["completion"], total])
        totals["all"] += total
        if attempt > 1:
            totals["retry"] += total
    ratio = totals["retry"] / totals["all"] if totals["all"] else 0.0
    print(f"# unparsed_events={bad}", file=sys.stderr)
    print(f"# total_tokens={totals['all']} retry_overhead={ratio:.2f}", file=sys.stderr)

if __name__ == "__main__":
    main()

Run it and hand-check two rows against the raw log. If unparsed_events is above zero, you have a logging bug, not a model bug — that distinction alone is worth the whole lab.

The free server changes the checkpoint question

On a box you own, the working directory is durable. On a free server, assume it is not. So the lab forces one habit:

# before you close the tab, before you retry anything:
tar czf ledger_$(date +%s).tgz run.jsonl ledger.csv budget_contract.png
sha256sum ledger_*.tgz >> ledger_hashes.txt

Two files leave the box: the archive and its hash. The hash matters because a grade you can't verify is a rumor. If the box dies mid-run, you still hold the evidence of what was spent.

Rubric: 100 points

Weight	Criterion	What earns full marks
25	Task oracle	A test that fails before and passes after, reproducible by a classmate
25	Ledger integrity	Every call logged, unparsed_events=0, hand-checked rows match
15	Retry discipline	Each retry has an attempt >= 2 line and a non-empty reason
15	Budget verdict	Run reported as within or over the screenshot contract, with a number
10	Replayability	A classmate reruns the archive and gets the same ledger totals
10	Failure write-up	One paragraph on what was not measured

No points exist for "it felt fast." No points exist for a pretty demo. A run that passes the test but blows the budget scores below a run that fails the test and reports the overspend honestly — that's the point of the lab.

Failure modes students hit every cohort

• Retry amnesia. The fix lands on attempt 4, but only attempt 4 is logged. Total cost now reads as one call. This is the single most common way a ledger lies.
• Reason inflation. reason: "retry" on every line is technically non-empty and completely useless. Grade it as unlabeled.
• Hash drift. The ledger is edited after the run "to fix a column name." The hash proves it. Regenerate, don't patch.
• Oracle smuggling. The test is weakened so the run finishes early. Compare the test file before and after; a shrinking assertion count is a fail.
• Budget cliff. The task passes at attempt 9, after 8 identical failures. Log it and move on — you've just learned more than a passing run would teach.

Stretch goals

1. Add a --fail-over flag to ledger.py that exits nonzero when total_tokens exceeds a value you pass in. Wire it into your run script.
2. Split retry_overhead by failure class (test_failed, timeout, tool_error) and see which class actually eats your allowance.
3. Write the same task twice, once stopping at attempt 2 and once at a budget cap, and compare which one a classmate finds more useful.
4. Add a second, independent check of the ledger — a shell one-liner using jq that sums the same fields. Two counters that agree beat one counter you trust.

What this lab can't tell you, and who should skip it

This lab says nothing about latency, throughput ceilings, or whether a model is good at your domain. It measures one thing: whether you can account for what a run consumed. Also, don't bother if:

• your workload can't leave your own network, so an external free tier is off the table;
• you're on committed paid capacity where the budget genuinely doesn't bind;
• or the grader never looks at your log, in which case metering is theater and you should spend the time on the oracle instead.

Numbers in your report are yours, not mine. Don't quote a total you didn't compute from your own JSONL — and re-check the free tier's current terms the day you run it, because allowances change and your screenshot is the only budget contract that counts.

If you run this, the useful write-up isn't the passing test. It's the ledger line where you realized the retries, not the model, were the expensive part.