Retry Budgets Are Real: Pairing Lessons from a Free LLM Endpoint

A retry loop without a budget is a quota leak waiting to happen. In a two-hour pairing session, a senior engineer forced a closer look at that assumption, and the surviving design was a SQLite-backed ledger that turns every retry into a recorded, budgeted decision. This article reconstructs the questions, the dead ends, and the exact artifact the pair kept.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The pairing session used MonkeyCode's free model access and free server option to simulate a realistic quota-constrained endpoint; the operator describes the current free tier as a 10 million token allowance and a free server.

The original problem was a batch summarizer that called a free LLM endpoint for each chunk of text. Occasionally the endpoint returned a 503 or 429, and the original implementation retried each chunk up to five times with a fixed backoff. That seemed safe until a particularly unlucky batch consumed an entire day's token allowance in one run.

The Senior's First Question: "What Is a Retry Worth?"

The question stopped the room. The immediate answer, "one token per attempt," ignored the opportunity cost: every retry consumed a slice of the shared monthly budget. The pair tried to track token usage per attempt, but the expected usage field at the endpoint was sometimes absent, especially on server errors. That led to a simpler but dangerous design: a constant retry limit of three. A constant limit fails because it does not distinguish between a long context task, already expensive, and a short one. One short task could retry three times and spend almost nothing, while a long one could exceed its cost threshold before the first retry.

Dead End One: An In-Memory Dictionary

The first concrete attempt was a Python dictionary mapping task IDs to attempt counters. It worked in a single process and looked clean in a demo. Then the pair ran the batch with two workers on the free server, and the dictionary failed silently: because each worker had its own memory, the same task could be retried by both workers simultaneously, effectively doubling the quota burn. The snippet below shows the naive shape:

attempts = {}
def should_retry(task_id, max_attempts):
    return attempts.get(task_id, 0) < max_attempts

A thread-safety feature like a global lock fixes the single-host case, but the free server option here means a single small VM, so a process-local dict should theoretically work. The real issue was more subtle: the pair forgot that the endpoint's SDK performs its own internal retries before the application-level logic ever runs. Those SDK-level retries bypass the dict entirely.

Dead End Two: A JSON File as a Shared Ledger

To capture all attempts, the pair moved the counters to a JSON file on the same server. A threading.Lock protected writes, and the batch ran successfully for two hours. Then they killed the process mid-flight to simulate a crash, and the log file became corrupted because the write was truncated. That failure resurrected an old rule: never hand-roll atomic file updates if a database is available.

The Kept Decision: SQLite as a Retry Ledger

The final design uses SQLite in WAL mode. It gives atomic increments, transaction isolation, and a single file that can be backed up without a second service. The workflow has four steps:

1. Create a retries table with task_id, attempts, retry_at, and token_budget.
2. Before each request, call can_retry(task_id, max_attempts).
3. After each response, call register_attempt(task_id, token_cost, max_tokens).
4. Sleep until retry_at if the attempt failed.

A compact implementation looks like this:

import sqlite3, time

class RetryLedger:
    def __init__(self, db_path="retry.db"):
        self.conn = sqlite3.connect(db_path, check_same_thread=False)
        self.conn.execute("PRAGMA journal_mode=WAL")
        self.conn.execute("""
            CREATE TABLE IF NOT EXISTS retries (
                task_id TEXT PRIMARY KEY,
                attempts INTEGER NOT NULL DEFAULT 0,
                retry_at REAL NOT NULL DEFAULT 0,
                token_budget INTEGER NOT NULL DEFAULT 0
            )
        """)
        self.conn.commit()

    def can_retry(self, task_id, max_attempts):
        row = self.conn.execute(
            "SELECT attempts, retry_at, token_budget FROM retries WHERE task_id=?",
            (task_id,)
        ).fetchone()
        if row is None:
            return True
        attempts, retry_at, budget = row
        return attempts < max_attempts and budget > 0 and time.time() >= retry_at

    def register_attempt(self, task_id, token_cost, max_tokens, backoff=1.5):
        with self.conn:
            row = self.conn.execute(
                "SELECT attempts, token_budget FROM retries WHERE task_id=?",
                (task_id,)
            ).fetchone()
            attempts, budget = row if row else (0, max_tokens)
            attempts += 1
            budget -= token_cost
            retry_at = time.time() + backoff
            self.conn.execute(
                "INSERT OR REPLACE INTO retries VALUES (?, ?, ?, ?)",
                (task_id, attempts, retry_at, budget)
            )
            return attempts, budget

Notice that token_budget is decremented on every attempt, including successful ones. That makes the budget a true cap on the total token volume spent on a single task, not just on retries.

Reproducible Test: Concurrency Cannot Exceed the Budget

The test below spawns many threads that hammer the same ledger with the same task ID. The invariant is simple: after all threads finish, the recorded token_budget must never go negative, and the number of attempts cannot exceed the configured maximum.

import threading
from retry_ledger import RetryLedger

def test_concurrent_budget_enforced():
    ledger = RetryLedger(":memory:")
    max_attempts = 3
    max_tokens = 50
    errors = []

    def worker(task_id):
        try:
            while ledger.can_retry(task_id, max_attempts):
                attempts, budget = ledger.register_attempt(
                    task_id, token_cost=10, max_tokens=max_tokens
                )
                assert budget >= 0
        except Exception as exc:
            errors.append(exc)

    threads = [
        threading.Thread(target=worker, args=("task-a",)) for _ in range(20)
    ]
    for t in threads:
        t.start()
    for t in threads:
        t.join()

    assert not errors
    final_row = ledger.conn.execute(
        "SELECT attempts, token_budget FROM retries WHERE task_id='task-a'"
    ).fetchone()
    assert final_row[0] <= max_attempts
    assert final_row[1] >= 0

Run it with pytest after placing the ledger class in a module named retry_ledger.py. The test may take a few seconds because each retry calls time.sleep(1.5) via the backoff, but the budget assertion is what matters.

Limitations and Who Should Skip This

This SQLite ledger is not a universal retry solution. It works on a single host or a small free server because SQLite is file-based; running multiple serverless instances against the same database file will cause locking contention. In that case, a Postgres or Redis-based counter is a better fit. The ledger also counts only the application-level retries; SDK-level retries and connection retries remain outside its scope. Teams that need millisecond-level retry decisions or that handle thousands of tasks per second should seek a purpose-built rate limiter. Finally, the ledger cannot judge response quality; a 200 with a truncated body is still counted as a success unless separate validation is added.

For a free-tier LLM endpoint with a hard token allowance, the pairing session made one point permanent: automatic retries are a budget decision, not a reliability mechanism. The SQLite ledger is the smallest piece of infrastructure that makes that decision auditable, and it fits comfortably inside a free server's constraints.