Receipt or It Didn't Happen: A Bootcamp Lab on Agent Token Budgets

The grade is not the final string. The grade is the receipt. If your agent answered the ticket and you cannot show how many tokens it burned, you did not finish the lab.

Correctness without a budget is a demo. Demos get applause. Demos also retry until the model is lucky. On a laptop that habit is sloppy. On a shared box it is rude. You starve the next student. You hide cost. You ship vibes.

This lab freezes one rule: no receipt, no points. Not because tokens are morally interesting. Because an agent you cannot meter is an agent you cannot grade twice.

Why this lab exists

I keep seeing the same failure in bootcamp agent homework. The student pastes a giant system prompt. The loop retries five times. The answer finally looks right. The PR is green. Nobody counted a single token.

Ask a mean question. Did they solve the task, or did they buy the task with retries?

Loop-call caps are not enough. Twelve tool calls can still dump a novel into the prompt every round. Runtime contracts are not enough. The JSON can be valid and still cost a fortune. A second machine can still be wasteful if both machines are free and nobody is watching the meter.

So we grade the meter.

What you will build

A tiny agent that must answer one fixture task under a hard course budget. The agent writes receipt.json. The grader reads that file first. The answer is checked second.

If the receipt is missing, malformed, over budget, or leaking secrets, the lab is a zero. Yes, even if the answer is perfect. That is the point.

Labeled as a course fixture, not a production billing system:

• Budget: 8000 estimated tokens for the whole run
• Max model calls: 4
• Task: refund-policy-v1 (a short policy question with a known oracle)
• Estimator: UTF-8 bytes divided by 4, rounded up

That estimator is a lab ruler, not the vendor tokenizer. We use one ruler so two students can be compared. We do not pretend it matches a real invoice.

Setup

Work in an isolated directory. If you are on a shared host, make the directory yours. Do not write into /tmp/agent. Someone else is already there.

python3 --version   # 3.11+
mkdir -p "$HOME/labs/token-receipt/$USER"
cd "$HOME/labs/token-receipt/$USER"
python3 -m venv .venv
source .venv/bin/activate
pip install -U pip requests python-dotenv
printf '.venv\n.env\nreceipt.json\n__pycache__/\n' > .gitignore

Create .env locally. Never commit it. The grader will fail you if a secret shows up in the receipt.

cat > .env << 'EOF'
LLM_BASE_URL=http://127.0.0.1:8080/v1
LLM_API_KEY=replace-me
LLM_MODEL=course-default
STUDENT_ID=changeme
TOKEN_BUDGET=8000
EOF

LLM_MODEL=course-default is a placeholder. Pin whatever endpoint your instructor hands you. Do not invent a brand name in the writeup. The receipt stores the raw string you actually called.

Need a shared endpoint so the whole cohort hits the same URL? I run this lab against MonkeyCode when I want free model access and a free server option instead of twelve personal API keys. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am not claiming a quota, a GPU, a model name, or that the free tier lasts forever. If the class endpoint moves, you change .env. The receipt still has to add up.

The artifact: a receipt-first harness

Four files. Copy them. Then break them on purpose.

tokens.py

# Course estimator. Not a vendor tokenizer. Not a billing API.
from __future__ import annotations


def estimate_tokens(text: str) -> int:
    if not text:
        return 0
    nbytes = len(text.encode("utf-8"))
    return max(1, (nbytes + 3) // 4)

Why bytes, not len(text)? Because a student once stuffed emoji into the system prompt and called it "short." Characters lie. Bytes lie less.

receipt.py

from __future__ import annotations

import json
import os
import re
import socket
from dataclasses import asdict, dataclass, field
from datetime import datetime, timezone
from pathlib import Path
from typing import Any

SECRET_RE = re.compile(r"(api[_-]?key|token|secret|password)\s*[:=]\s*\S+", re.I)


@dataclass
class Receipt:
    student_id: str
    hostname: str
    pid: int
    started_at: str
    ended_at: str | None = None
    task_id: str = "refund-policy-v1"
    model: str = ""
    base_url_host: str = ""
    prompt_tokens: int = 0
    completion_tokens: int = 0
    calls: int = 0
    budget: int = 8000
    final_answer: str = ""
    over_budget: bool = False
    notes: list[str] = field(default_factory=list)

    @property
    def total_tokens(self) -> int:
        return self.prompt_tokens + self.completion_tokens

    def add_call(self, prompt: str, completion: str, estimate_tokens) -> None:
        self.calls += 1
        self.prompt_tokens += estimate_tokens(prompt)
        self.completion_tokens += estimate_tokens(completion)
        self.over_budget = self.total_tokens > self.budget

    def seal(self, answer: str) -> None:
        self.final_answer = answer.strip()
        self.ended_at = datetime.now(timezone.utc).isoformat()
        self.over_budget = self.total_tokens > self.budget

    def to_json(self) -> dict[str, Any]:
        data = asdict(self)
        data["total_tokens"] = self.total_tokens
        return data


def new_receipt(budget: int, model: str, base_url: str) -> Receipt:
    host = ""
    try:
        # Store host only. Never store the key, never store the full URL.
        host = base_url.split("://", 1)[-1].split("/", 1)[0]
    except Exception:
        host = "unknown"
    return Receipt(
        student_id=os.environ.get("STUDENT_ID", "unknown"),
        hostname=socket.gethostname(),
        pid=os.getpid(),
        started_at=datetime.now(timezone.utc).isoformat(),
        budget=budget,
        model=model,
        base_url_host=host,
    )


def write_receipt(path: Path, receipt: Receipt) -> None:
    payload = json.dumps(receipt.to_json(), indent=2) + "\n"
    if SECRET_RE.search(payload):
        raise ValueError("refusing to write a receipt that looks like it contains a secret")
    path.write_text(payload, encoding="utf-8")

Notice what is missing. No API key. No full URL. No raw prompt dump. A flight recorder is a different lab. This one is a cash register.

agent_lab.py

from __future__ import annotations

import os
import sys
from pathlib import Path

import requests
from dotenv import load_dotenv

from receipt import new_receipt, write_receipt
from tokens import estimate_tokens

TASK_ID = "refund-policy-v1"
QUESTION = (
    "A customer bought a cable 12 days ago. It still works. "
    "They want a refund because the color is wrong. "
    "Policy: refunds within 14 days for unused items; color preference is allowed. "
    "Reply with exactly YES or NO, then a one-line reason."
)
SYSTEM = (
    "You are a refund clerk. Use only the policy in the user message. "
    "Do not invent extra rules. Keep the reply under 40 words."
)


def chat(base_url: str, api_key: str, model: str, messages: list[dict]) -> str:
    url = base_url.rstrip("/") + "/chat/completions"
    headers = {"Authorization": f"Bearer {api_key}"}
    body = {
        "model": model,
        "messages": messages,
        "temperature": 0,
        "max_tokens": 80,
    }
    resp = requests.post(url, json=body, headers=headers, timeout=60)
    resp.raise_for_status()
    data = resp.json()
    return data["choices"][0]["message"]["content"]


def main() -> int:
    load_dotenv()
    base_url = os.environ["LLM_BASE_URL"]
    api_key = os.environ["LLM_API_KEY"]
    model = os.environ.get("LLM_MODEL", "course-default")
    budget = int(os.environ.get("TOKEN_BUDGET", "8000"))
    out = Path("receipt.json")

    rec = new_receipt(budget, model, base_url)
    messages = [
        {"role": "system", "content": SYSTEM},
        {"role": "user", "content": QUESTION},
    ]
    answer = ""
    try:
        for attempt in range(1, 5):
            if rec.total_tokens >= budget:
                rec.notes.append("stopped before call: budget exhausted")
                break
            prompt_blob = SYSTEM + "\n" + QUESTION + "\n" + answer
            completion = chat(base_url, api_key, model, messages)
            rec.add_call(prompt_blob, completion, estimate_tokens)
            answer = completion.strip()
            if answer.upper().startswith("YES") or answer.upper().startswith("NO"):
                rec.notes.append(f"accepted on attempt {attempt}")
                break
            messages.append({"role": "assistant", "content": completion})
            messages.append(
                {"role": "user", "content": "Format invalid. Reply YES or NO, then one line."}
            )
            rec.notes.append(f"retry {attempt}: missing YES/NO prefix")
        rec.seal(answer)
        write_receipt(out, rec)
    except Exception as exc:
        rec.notes.append(f"error: {type(exc).__name__}")
        rec.seal(answer)
        try:
            write_receipt(out, rec)
        except Exception:
            pass
        print(f"failed: {exc}", file=sys.stderr)
        return 1

    print(f"answer={rec.final_answer!r}")
    print(f"tokens={rec.total_tokens}/{rec.budget} calls={rec.calls}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Four calls max. Eighty completion tokens max per call. Temperature zero. That is not because temperature zero is magic. It is because lucky retries are how students blow a shared budget and then swear the code is deterministic.

grade.py

from __future__ import annotations

import json
import os
import re
import sys
from pathlib import Path

REQUIRED = {
    "student_id",
    "hostname",
    "pid",
    "started_at",
    "ended_at",
    "task_id",
    "model",
    "base_url_host",
    "prompt_tokens",
    "completion_tokens",
    "total_tokens",
    "calls",
    "budget",
    "final_answer",
    "over_budget",
}
SECRET_RE = re.compile(r"(api[_-]?key|sk-|bearer\s+\S+)", re.I)


def fail(msg: str) -> int:
    print(f"FAIL: {msg}")
    return 1


def main() -> int:
    path = Path("receipt.json")
    if not path.exists():
        return fail("receipt.json missing")
    try:
        data = json.loads(path.read_text(encoding="utf-8"))
    except json.JSONDecodeError:
        return fail("receipt.json is not JSON")

    missing = REQUIRED - set(data)
    if missing:
        return fail(f"missing fields: {sorted(missing)}")

    raw = path.read_text(encoding="utf-8")
    if SECRET_RE.search(raw):
        return fail("receipt looks like it leaked a secret")

    if data.get("task_id") != "refund-policy-v1":
        return fail("wrong task_id")
    if not data.get("student_id") or data["student_id"] == "changeme":
        return fail("set STUDENT_ID")
    if int(data["calls"]) < 1:
        return fail("zero model calls")
    if int(data["completion_tokens"]) < 1:
        return fail("zero completion tokens; empty model output is not a solution")
    if bool(data["over_budget"]) or int(data["total_tokens"]) > int(data["budget"]):
        return fail(f"over budget: {data['total_tokens']}/{data['budget']}")
    if int(data["total_tokens"]) != int(data["prompt_tokens"]) + int(data["completion_tokens"]):
        return fail("total_tokens does not equal prompt + completion")

    answer = str(data["final_answer"]).strip().upper()
    if not answer.startswith("YES"):
        return fail("oracle expected YES for unused item inside 14 days, color preference allowed")

    print("PASS")
    print(f"tokens={data['total_tokens']}/{data['budget']} calls={data['calls']}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Run it like a student, then like a grader.

python agent_lab.py
python grade.py
cat receipt.json

If grade.py prints PASS and you never opened receipt.json, you still failed the lab in spirit. Read the file. That is the whole exercise.

Checkpoints

Do these in order. Do not skip to stretch goals because the model sounded confident.

1. Receipt exists after failure. Kill the network on purpose (LLM_BASE_URL=http://127.0.0.1:1) and confirm receipt.json still appears with an error note. A crash is not an excuse to skip the meter.
2. Budget is a hard stop. Temporarily set TOKEN_BUDGET=50 and confirm the grader fails with over budget. If your agent keeps calling the model anyway, you built a diary, not a budget.
3. Oracle still matters. Change the fixture in your head: used item, day 15. You should get NO. If you only test the happy path, you are grading luck again.
4. No secrets in the artifact. Put LLM_API_KEY=sk-this-is-fake in .env, run a successful pass, then grep -Ei 'sk-|api_key' receipt.json must be empty.
5. Shared-host isolation. Print hostname and pid from the receipt. Two students on one box must not share a working directory. If your files land in /tmp, start over.

Checkpoint 2 is the one people argue with. "But my answer was right." Cool. The lab is not "was the refund allowed." The lab is "can you finish inside the meter."

Fair grading rubric

Total: 20 points. Partial credit is allowed only where the receipt is valid.

Points	What I actually score
4	receipt.json present, valid JSON, all required fields
3	total_tokens == prompt_tokens + completion_tokens, over_budget consistent
3	Under the published budget; grader PASS on the fixture
3	Oracle: YES plus a one-line reason, no extra essay
3	No secrets, no full URL, no key material in the receipt
2	Isolation: unique workdir, STUDENT_ID set, hostname/pid filled
2	Checkpoint 1: receipt still written when the endpoint is down

Automatic zeros:

• Missing receipt
• Over budget
• Secret-shaped strings in receipt.json
• STUDENT_ID=changeme
• Editing grade.py to print PASS

I do not grade prose style. I do not grade how "agentic" the logs feel. I grade whether you can repeat the run without lighting the shared endpoint on fire.

Stretch goals

Only after a clean PASS.

• Prompt cache. Hash SYSTEM + QUESTION. If the hash hits a local cache.json, do not spend tokens. Add cache_hits to the receipt. Stretch grader: fail if the second run of the same task makes a model call.
• Two students, one box. Open two directories with two STUDENT_IDs. Run in parallel. Neither receipt may be overwritten. If you used a fixed receipt.json in /tmp, you will learn this the loud way.
• Shrink the system prompt. Keep the oracle. Cut SYSTEM until estimated prompt tokens drop without changing the answer. Record before/after in notes. Do not claim a vendor cost savings. You only have the course ruler.
• Fail closed on empty completions. If completion_tokens == 0, treat it as a platform miss, not a student success. The starter grader already does this. Keep that behavior if you rewrite the loop.

What this is not

This is not a model bake-off. We are not publishing tokens-per-second. We are not naming a champion vendor. We are not pretending byte/4 equals a production tokenizer.

It is also not a security boundary. A free shared server is still a shared server. Other processes exist. Disk is not private because your folder name is cute. If the homework ever includes real customer tickets, stop. Use synthetic fixtures.

Who should skip this approach?

• Anyone billing real customers with this estimator. You will be wrong.
• Anyone who needs a secret store, VPC, or audit log. .env on a class box is a convenience, not a control.
• Anyone whose "agent" is a single hardcoded if with no model call. There is nothing to meter. Pick a different lab.
• Instructors who want to grade only the final sentence. You will train retry-until-green. Please do not.

The uncomfortable lesson

Free inference makes this worse before it makes it better. When the model feels free, students write novels into the system prompt. When the server feels free, they leave processes running. The receipt is how you make "free" visible.

Would I ship this harness to production? No. Would I accept an agent PR that cannot show its token receipt? Also no.

Run the fixture. Break the budget. Read the JSON. If you want the cohort on one endpoint instead of a pile of personal keys, MonkeyCode's free model access and free server option is the setup I use for this lab. Fork the harness, set TOKEN_BUDGET lower than you think you need, and see which prompt you actually have to delete.