Reviewing an Agent PR That Adds a Token Budget Ledger

A reviewer opens a pull request at 09:12. The title reads feat: add token budget ledger. The diff is 180 lines and the CI badge is green.

That is the moment to slow down. A ledger is accounting code. Everyone downstream trusts it, and almost nobody verifies it.

The PR in this review came from an agent session run on MonkeyCode. The project advertises free model access and a free server option.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The workflow below stays useful with every product mention removed. It is a review pattern, not a vendor tour.

1. Write down what the diff claims

Three claims usually hide inside a ledger PR. Each claim needs its own test.

1. The ledger counts real usage, not an estimate.
2. The ledger blocks calls that would exceed the budget.
3. The ledger survives retries, cancels, and dropped connections.

Claim one is a data question. Claim two is a policy question. Claim three is a state question.

Most agent PRs get claim two right. Claims one and three fail quietly in production.

2. Split the diff into trust, revert, and test

Use a fixed triage table before reading line by line. The table stops reviewers from grading style while missing arithmetic.

Diff area	Decision	Reason
usage parsed from the response body	Trust, then test	Server usage is the only authoritative number
Local tokenizer estimate used for enforcement	Revert	Two tokenizers, two answers, one budget
In-memory spent counter	Revert	Restarts and replicas reset it
Budget check before dispatch	Test	Off-by-one on > versus >=
Retry wrapper	Test	Retries are where double charging appears
Logging and metrics fields	Trust	Noise, not correctness

Revert first, argue later. A reverted estimate is cheaper than a wrong invoice.

3. Make the model server deterministic

Reconciliation needs fixed numbers. Point the client at a local server that always returns the same usage object.

# fake_model.py - no network egress, fixed usage, two modes
import json, sys
from http.server import BaseHTTPRequestHandler, HTTPServer

USAGE = {"prompt_tokens": 812, "completion_tokens": 96}
MODE = sys.argv[1] if len(sys.argv) > 1 else "normal"

class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        self.rfile.read(int(self.headers.get("content-length", 0)))
        payload = {"id": "resp_fixed_1", "choices": [{"message": {"content": "ok"}}]}
        if MODE != "no_usage":
            payload["usage"] = USAGE
        body = json.dumps(payload).encode()
        self.send_response(200)
        self.send_header("content-type", "application/json")
        self.send_header("content-length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, *args):
        pass

if __name__ == "__main__":
    HTTPServer(("127.0.0.1", 8099), Handler).serve_forever()

Start it with python fake_model.py. Start it again with python fake_model.py no_usage for the fail-closed test.

Two modes are enough. More modes mean the harness is testing itself.

4. Reconcile the ledger against server usage

This test catches most of the damage. The ledger must equal the sum of server usage, exactly.

# test_ledger_reconcile.py
import pytest
from ledger import Ledger, call_model, BudgetExceeded, TransientError, MissingUsage

URL = "http://127.0.0.1:8099"

def test_single_call_matches_server_usage():
    ledger = Ledger(budget=10_000)
    resp = call_model(URL, "hello", ledger=ledger)
    assert resp["usage"]["prompt_tokens"] == 812
    assert ledger.spent == 812 + 96
    assert ledger.remaining == 10_000 - 908

Run the same assertion against the reverted estimate. If the estimate agrees by luck, the test still passes. That luck is not a design.

Add a boundary test next. The call that lands exactly on the limit must be allowed once.

def test_exact_budget_boundary():
    ledger = Ledger(budget=908)
    call_model(URL, "hello", ledger=ledger)
    assert ledger.remaining == 0
    with pytest.raises(BudgetExceeded):
        call_model(URL, "hello", ledger=ledger)

5. Test retries as a charging problem

A retry wrapper turns one logical request into several HTTP calls. The ledger must charge once per logical request, not once per attempt.

def test_retry_charges_once_per_request_id():
    ledger = Ledger(budget=10_000)
    call_model(URL, "hello", ledger=ledger, request_id="r-1")
    with pytest.raises(TransientError):
        call_model(URL, "hello", ledger=ledger, request_id="r-1")  # injected drop
    assert ledger.spent == 908

If this fails, revert the counter and keep the request id. Idempotency keys belong in the ledger key space, not in a comment.

Cancelled streams need the same treatment. A cancelled stream still bills for tokens already emitted.

6. Fail closed when usage is missing

Restart the fake server in no_usage mode. The ledger must refuse to guess.

def test_missing_usage_fails_closed():
    ledger = Ledger(budget=10_000)
    with pytest.raises(MissingUsage):
        call_model(URL, "hello", ledger=ledger)
    assert ledger.spent == 0          # nothing charged, nothing invented
    assert ledger.state == "blocked"

Charging zero and continuing is the most common bug in this class. It turns a budget into a suggestion.

7. Read the failure table before approving

Symptom	Root cause	Review action
Spend drifts upward on long runs	Estimate used for enforcement	Revert to server usage
Budget resets after deploy	In-memory counter	Move state out, or document the reset
Double charge on timeout	Retry without request id	Revert retry, add idempotency key
Free calls after an upstream change	Missing usage treated as zero	Fail closed
One replica overspends	Per-process counters	Centralise or shard the budget

Two rows are state bugs. One is a data bug. All three reach production when CI only checks the happy path.

Limitations and who should skip this

The harness proves arithmetic and idempotency. It proves nothing about upstream pricing, provider quotas, or clock skew between replicas.

Skip this workflow for batch jobs that reserve a fixed budget up front. Skip it when nobody enforces a budget at all. A ledger nobody reads is dead code with tests attached.

Skip it too if the team cannot run a six-line local server. That constraint deserves fixing first.

Closing

A green badge on an agent PR means the agent's own tests passed. It does not mean the numbers are right.

Revert the estimate, keep the server usage, and pin the boundary test. Then the ledger earns its place in the diff.

Teams that want to rehearse this harness without wiring a paid provider can start on MonkeyCode's free model access and free server option. Verify the current terms before relying on them for anything long-lived.