Learn Why Agent Token Counts Mislead by Building a Tiny Budget Harness

A few days ago I read a discussion on DEV arguing that sub-agent metrics are not comparable to main-thread metrics. My first reaction was: "surely tokens are tokens?" So I built the smallest possible experiment to prove myself wrong.

Learning question: if I send the same task to a chat model twice, will the token counts match — and if not, what exactly is driving the difference?

By the end you will have a ~70-line, standard-library-only Python harness that calls any OpenAI-compatible chat endpoint, logs token usage per call, and enforces a hard budget. Then we will use it to show why "agent A used 4,000 tokens, agent B used 6,000" is not a meaningful comparison on its own.

Prerequisites

• Python 3.10+ (I used 3.12; no third-party packages)
• Access to any OpenAI-compatible chat completions endpoint. I used MonkeyCode, which offers free model access and a free server option, so the whole experiment cost me nothing — but any compatible endpoint works.
• A rough idea of what a "token" is (a chunk of text the model bills and reasons over — words or word pieces)

Step 1 — The harness

Save as budget_harness.py:

#!/usr/bin/env python3
"""Tiny token-budget harness for OpenAI-compatible chat endpoints."""
import json
import os
import sys
import urllib.request

BASE_URL = os.environ.get("CHAT_BASE_URL", "https://YOUR-ENDPOINT/v1")
API_KEY = os.environ.get("CHAT_API_KEY", "")
MODEL = os.environ.get("CHAT_MODEL", "your-model-name")

class BudgetExceeded(Exception):
    pass

class BudgetHarness:
    def __init__(self, token_budget: int):
        self.budget = token_budget
        self.used = 0
        self.log = []  # visible intermediate state: every call, every count

    def chat(self, messages: list[dict], label: str) -> str:
        body = json.dumps({
            "model": MODEL,
            "messages": messages,
            "temperature": 0,
        }).encode()
        req = urllib.request.Request(
            f"{BASE_URL}/chat/completions",
            data=body,
            headers={
                "Content-Type": "application/json",
                "Authorization": f"Bearer {API_KEY}",
            },
        )
        with urllib.request.urlopen(req, timeout=60) as resp:
            data = json.loads(resp.read())

        usage = data.get("usage", {})
        total = usage.get("total_tokens", 0)
        self.used += total
        self.log.append({
            "label": label,
            "prompt_tokens": usage.get("prompt_tokens"),
            "completion_tokens": usage.get("completion_tokens"),
            "total_tokens": total,
        })
        if self.used > self.budget:
            raise BudgetExceeded(
                f"budget {self.budget} exceeded at call '{label}' (used {self.used})"
            )
        return data["choices"][0]["message"]["content"]

    def report(self) -> None:
        print(f"{'call':<22}{'prompt':>8}{'completion':>12}{'total':>8}")
        for row in self.log:
            print(f"{row['label']:<22}{row['prompt_tokens']:>8}"
                  f"{row['completion_tokens']:>12}{row['total_tokens']:>8}")
        print(f"\nBudget: {self.budget} | Used: {self.used} "
              f"| Remaining: {self.budget - self.used}")

Nothing fancy here on purpose. The two things that matter: we read the usage field from every response (most OpenAI-compatible servers return it), and we keep a visible running log instead of a single final number. The log is the lesson.

Step 2 — The experiment: one task, three framings

Save as experiment.py:

from budget_harness import BudgetHarness, BudgetExceeded

TASK = "Explain what a hash collision is in two sentences."

h = BudgetHarness(token_budget=4000)

try:
    # Call 1: bare task
    h.chat([{"role": "user", "content": TASK}], label="bare-task")

    # Call 2: same task, polite system prompt
    h.chat([
        {"role": "system", "content": "You are a patient CS tutor. " * 20},
        {"role": "user", "content": TASK},
    ], label="with-system-prompt")

    # Call 3: same task, but with fake conversation history (like an agent would carry)
    history = [{"role": "user", "content": f"Earlier question {i}: recap."} for i in range(15)]
    history.append({"role": "user", "content": TASK})
    h.chat(history, label="with-history")
except BudgetExceeded as e:
    print("STOPPED:", e)

h.report()

Run it:

export CHAT_BASE_URL="https://your-endpoint/v1"
export CHAT_API_KEY="your-key"
export CHAT_MODEL="your-model-name"
python experiment.py

Expected output (illustrative — your numbers will differ)

call                    prompt   completion   total
bare-task                  21           47      68
with-system-prompt        108           47     155
with-history              356           52     408

Budget: 4000 | Used: 631 | Remaining: 3369

Do not worry if your exact counts differ — tokenizers and models vary, and that's fine. What should not vary is the shape of the result.

What actually happened

The task — the part a human would describe as "the work" — is identical in all three calls. The completion is roughly the same size every time. But the prompt tokens grew by ~5x and then ~17x, purely because of context I wrapped around the task.

This is the trap behind "my agent used 4k tokens and yours used 6k":

1. Prompt tokens dominate agent workloads. An agent that carries conversation history, tool outputs, or a long system prompt re-pays for all of it on every single call. Two agents doing the same task can differ by an order of magnitude in billed tokens depending on how much context each one drags along.
2. Total tokens hide the split. If the report only showed Used: 631, you could not tell a cheap-task-expensive-context run from an expensive-task run. Always log prompt_tokens and completion_tokens separately.
3. A budget is a correctness tool, not just a cost tool. The BudgetExceeded exception forces you to decide in advance how much an experiment is allowed to cost. For a student on free tiers, this is the difference between a fun afternoon and a dead API key.

Try the failing case yourself

Before running, predict: with token_budget=200 and the same three calls, which call trips the budget — with-system-prompt or with-history? Then lower the budget to 100 and predict again. If your prediction was wrong, post your numbers in the comments; I'd genuinely like to see a minimal case where the ordering flips (a model with a very chatty completion style might do it).

Another good error input: point CHAT_BASE_URL at an endpoint that omits the usage field. The harness will happily record zeros and never trip the budget — a nice demonstration that a budget guard is only as honest as the telemetry it reads.

Where MonkeyCode fit in my run

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I ran this on MonkeyCode because, as a student, the two things I needed were (a) a model I could call without a credit card and (b) somewhere to run the script that isn't my laptop — and it currently offers both free model access and a free server option. The harness doesn't depend on it; any OpenAI-compatible endpoint with usage reporting works identically, and I deliberately kept the code provider-neutral so you can swap endpoints with one environment variable. If you're also learning on a tight budget, that combination is a convenient way to repeat this experiment — but verify the current free-tier terms yourself before relying on them for anything long-running.

Limitations and who should skip this

• This is not a benchmark. I ran each call a handful of times on one model. Token counts vary by model, tokenizer, and even API version; do not quote my illustrative numbers as measurements of any specific provider.
• usage is server-reported. Some endpoints omit it, cache it, or count differently. If you need billing-grade accuracy, check your provider's docs and reconcile against their dashboard.
• Temperature 0 reduces but doesn't remove variance. Some hosted models are non-deterministic even at temperature 0; expect small wobble in completion tokens.
• Skip this if you already run production LLM observability (LangSmith, OpenTelemetry GenAI semconv, etc.) — this harness is a learning tool, not monitoring infrastructure.

What you should understand now

• Why "tokens used" is a distribution shaped by context management, not a property of the task.
• Why comparing agents (or sub-agents vs. main thread) on total tokens alone is meaningless without the prompt/completion split and the context each side carried.
• How to wrap any experiment in a hard token budget using only the standard library.

Extension exercise: modify the harness to estimate cost per call by adding a per-model price table, then run call 3 with history lengths of 5, 15, and 50. Plot the curve — the slope is the real price of an agent with a long memory.