Reviewing an Agent PR That Retries Model Calls Without a Budget

A review request lands on a Tuesday morning. An agent has opened a PR titled "add retry to model client."

The diff is 41 lines. It touches one file. CI is green.

Most reviewers approve it in ninety seconds. Two spend bugs are sitting inside those lines.

The diff looks reasonable at first glance

The agent wrapped the model call in a retry loop. It added exponential backoff. It caught broad exceptions.

None of that is obviously wrong on its own. The problem is what the loop never tracks: a total budget.

This scenario is illustrative, not a report of a specific incident. The code below is a composite of retry loops that agents commonly generate.

Step 1: Classify the diff before you read it

Agent PRs mix three kinds of change. Each kind deserves a different review depth.

1. Mechanical edits: renames, formatting, import sorting. Skim these.
2. Local logic: one function, one file. Read fully, then test.
3. Resource control: retries, caches, locks, limits. Read adversarially.

Retry logic always lands in the third bucket. A retry loop is a spend multiplier and a load multiplier.

Step 2: Find the missing budget

Here is the shape agents usually produce.

async def call_model(client, prompt):
    for attempt in range(5):
        try:
            return await client.complete(prompt)
        except Exception:
            await asyncio.sleep(0.5 * 2 ** attempt)
    raise RuntimeError("model call failed")

Five attempts means five billable requests per call. The loop has no token ceiling. It has no wall-clock ceiling either.

Multiply that by a fan-out batch of 200 items. One bad afternoon becomes 1,000 requests.

The table below is arithmetic, not a benchmark. It assumes 1,200 tokens per attempt.

Attempts per call	Items	Requests	Tokens
1	200	200	240,000
3	200	600	720,000
5	200	1,000	1,200,000

A free allowance disappears faster than most reviewers expect. Any operator-reported free tier, including MonkeyCode's stated allowance of 10 million free tokens, should be treated as a finite resource in review.

Step 3: Check what the loop sleeps on

await asyncio.sleep yields control back to the event loop. time.sleep blocks the entire loop.

Agents mix the two constantly, especially in "fixed" revisions. Search the diff for time.sleep in any async path and revert on sight.

Also check for jitter. Retries without jitter synchronize across workers and rebuild the spike you were avoiding. Look for random.uniform near the delay calculation.

Step 4: Reject retries on non-retryable errors

except Exception retries a 400 validation error. It also retries auth failures and schema mismatches. That is pure waste with a clean bill attached.

Require a predicate instead.

RETRYABLE_STATUS = {408, 425, 429, 500, 502, 503, 504}

def is_retryable(exc: Exception) -> bool:
    status = getattr(exc, "status_code", None)
    if status is not None:
        return status in RETRYABLE_STATUS
    return isinstance(exc, (TimeoutError, ConnectionError))

This predicate is the highest-value three lines in the review. It removes retries that cannot succeed.

Step 5: Replace the loop with a bounded helper

A budget object makes the bound explicit and testable.

The helper below requires Python 3.11 or newer, because it uses asyncio.timeout.

import asyncio
import random
from dataclasses import dataclass

@dataclass
class Budget:
    max_tokens: int
    max_attempts: int = 3
    base_delay: float = 0.5
    spent_tokens: int = 0

    def remaining(self) -> int:
        return self.max_tokens - self.spent_tokens

    def charge(self, usage: int) -> None:
        self.spent_tokens += max(usage, 0)

    def can_retry(self, attempt: int) -> bool:
        return attempt < self.max_attempts and self.remaining() > 0

    def backoff(self, attempt: int) -> float:
        delay = self.base_delay * 2 ** (attempt - 1)
        return delay + random.uniform(0, delay * 0.1)


class ModelCallFailed(RuntimeError):
    pass


async def call_with_budget(client, prompt, budget: Budget, deadline_s: float = 20.0):
    async with asyncio.timeout(deadline_s):
        for attempt in range(1, budget.max_attempts + 1):
            if budget.remaining() <= 0:
                raise ModelCallFailed("token budget exhausted")
            try:
                resp = await client.complete(prompt, max_tokens=budget.remaining())
            except Exception as exc:
                if not is_retryable(exc):
                    raise
                budget.charge(getattr(exc, "usage", 0))
                if not budget.can_retry(attempt):
                    raise
                await asyncio.sleep(budget.backoff(attempt))
                continue
            budget.charge(resp.usage)
            return resp
        raise ModelCallFailed("no attempt succeeded")

Note the difference from the agent version. Failed attempts are charged too, when the provider reports usage on the error. Silent undercounting is a common agent bug.

Step 6: Prove the bound with a fake client

The tests below need no plugins. asyncio.run keeps the harness dependency-free.

import asyncio
import pytest
from app.budget import Budget, ModelCallFailed, call_with_budget


class ModelError(Exception):
    def __init__(self, status_code, usage=1200):
        super().__init__(f"status {status_code}")
        self.status_code = status_code
        self.usage = usage


class Response:
    def __init__(self, usage):
        self.usage = usage


class FakeClient:
    def __init__(self, statuses, usage=1200):
        self.statuses = list(statuses)
        self.usage = usage
        self.requests = 0

    async def complete(self, prompt, max_tokens):
        self.requests += 1
        status = self.statuses.pop(0) if self.statuses else 200
        if status != 200:
            raise ModelError(status, usage=self.usage)
        return Response(usage=self.usage)


def test_retry_stops_when_token_budget_runs_out():
    client = FakeClient([503, 503, 503, 503])
    budget = Budget(max_tokens=3000, max_attempts=5, base_delay=0.0)
    with pytest.raises(ModelError):
        asyncio.run(call_with_budget(client, "hi", budget))
    assert client.requests == 3
    assert budget.spent_tokens >= budget.max_tokens


def test_non_retryable_error_is_not_retried():
    client = FakeClient([400, 200])
    budget = Budget(max_tokens=10_000, max_attempts=5, base_delay=0.0)
    with pytest.raises(ModelError):
        asyncio.run(call_with_budget(client, "hi", budget))
    assert client.requests == 1


def test_deadline_caps_wall_clock_time():
    class SlowClient:
        async def complete(self, prompt, max_tokens):
            await asyncio.sleep(5)
            raise AssertionError("should never resolve")

    budget = Budget(max_tokens=10_000, max_attempts=5, base_delay=0.0)
    with pytest.raises(TimeoutError):
        asyncio.run(call_with_budget(SlowClient(), "hi", budget, deadline_s=0.05))

Run it with one command.

python -m pytest -q tests/test_budget.py

Three assertions carry the whole review. Request count is bounded. Non-retryable errors stop immediately. Wall-clock time is capped.

What to trust, what to revert, what to test

Signal in the diff	Verdict	Required evidence
except Exception around the model call	Revert	Replace with is_retryable predicate
time.sleep inside an async path	Revert	None; it blocks the event loop
Literal retry count, no budget object	Change requested	Test asserting a maximum request count
Failed attempts not charged	Change requested	Test with a usage-bearing error payload
asyncio.timeout present	Trust on 3.11+	Wall-clock assertion
Jittered backoff	Trust	Assertion that total delay stays under the deadline
Attempt number in logs	Trust	Check the log line exists on retry

Review the diff against this table before reading the prose description. Agent PR bodies often describe the intent, not the behavior.

Where this harness can run

Running the fake-client tests is free. Running them against hosted models is not.

The operator states that the project is open source, with free model access, a free server option, and an allowance of 10 million free tokens. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

For this review workflow, the hosted workspace matters more than the token allowance. A reviewer can re-run the harness on a branch without provisioning a local GPU. Treat license terms and quota details as operator-stated, and verify the current terms yourself before depending on them.

Limitations and who should not use this

• Budget accounting depends on provider-reported usage per response. If your provider omits usage on errors, count requests instead.
• Streaming responses often report usage only at the end. Without a final usage event, the token bound becomes approximate.
• asyncio.timeout needs Python 3.11 or newer. Adapt with asyncio.wait_for on older runtimes.
• The budget is client-side. It cannot stop billing for a request already in flight.
• Free tiers and quotas change. Do not encode a promotional allowance into production defaults.

Skip this approach if your provider already enforces hard spend caps, if your codebase is threaded and synchronous, or if you need billing-layer guarantees rather than client-layer ones.

Takeaway

A retry loop is a resource-control change, not a convenience change. Review it as a budget problem. Then assert the budget in a test before you approve.