Postmortem: The 2 A.M. LLM Retry Loop That Paid for Your Silence

Your phone wakes you at 2:13 a.m., and the alert says what the dashboard confirms seconds later: the summarizer agent has been running the same failing prompt since midnight. The job was supposed to read new email threads, compress them into three-line bullets, and file them into the team's wiki. One malformed message slipped past validation, the extraction step threw an exception, and the retry loop restarted the whole pipeline with the same broken input and an ever-growing context.

A postmortem is not a blame document, and it is not an apology letter either. It is a timeline, a list of contributing factors, and a durable fix that survives the next change you ship. This article walks through the incident shape that most teams running autonomous agents have already survived once. It ends with a routing strategy that stops the money leak without slowing down the jobs that actually matter.

The timeline

What follows is a representative night, not a single measured incident. At midnight the scheduler wakes the agent, and eight minutes later the first attempt fails on the malformed payload. Sixteen minutes in, the agent calls itself again and the context now includes the prior failure trace. By 1:40 the window is heavy and the retries continue at the same fixed interval, and by 2:13 the provider threshold alert fires and you are officially part of the story.

The line that hurts is not the first error but the silent amplification between retry and context. Each loop iteration carried a few more thousand tokens of history, produced nothing useful, and consumed your allowance as if it were doing real work. By morning the only artifact was a long charge list and a queue full of unread email.

Contributing factors

Three causes sit behind the visible failure. First, the retry function had no exponential backoff and no jitter, so every exception hit the provider at the same fixed cadence. Second, the agent had no token budget, meaning nothing in the loop asked how much capacity the current attempt had already spent. Third, the pipeline treated every task as equal and sent cheap classification and expensive reasoning to the same endpoint.

There is also the human factor, and it is worth naming. Because the agent produced plausible output, the review process became a rubber stamp, and nobody tested the loop under failure conditions. The reviewer was never themselves reviewed, so the loop was free to burn the night away.

The durable fix

Start with a router that decides which tier any task deserves before the prompt touches a model. Repetitive chores go to a free model, hard reasoning stays on your paid provider, and a daily token budget sits between the two. When the budget is exhausted the system degrades to the free tier instead of quietly overdrawing the account, and every retry uses jittered exponential backoff so failures spread out instead of clustering.

MonkeyCode fits this design because it offers free model access and a free server option, which covers exactly the two expenses that loop was burning through. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free model access fills the routing tier in this post, and the free server hosts the scheduler and the gateway so you are not renting a second cloud box for a job that only runs at night. Free tiers change, so verify the current allowance and model lineup on the project's repository before you depend on any specific number from this article.

A minimal gateway that enforces the pattern looks like this:

import random
import time
from dataclasses import dataclass

@dataclass
class Task:
    id: str
    prompt: str
    tier: str  # "cheap" or "frontier"

class BudgetedRouter:
    def __init__(self, daily_budget_tokens, free_client, paid_client):
        self.budget = daily_budget_tokens
        self.used = 0
        self.free_client = free_client   # MonkeyCode free model access
        self.paid_client = paid_client   # your existing provider

    def route(self, task):
        if task.tier == "cheap" or self.used + len(task.prompt) >= self.budget:
            client = self.free_client
        else:
            client = self.paid_client
        for attempt in range(3):
            try:
                result = client.complete(task.prompt, max_tokens=512)
                self.used += len(task.prompt) + 512
                return result
            except Exception:
                if attempt == 2:
                    raise
                time.sleep((2 ** attempt) + random.uniform(0, 1))

The edge that makes this safe is the budget check before the client selection, not after the call. You pay the small cost of a routing decision on every task, and in exchange you get a hard ceiling on what a single buggy loop can consume. Put that behind a simple entry point and the loop is contained no matter how confused the agent becomes.

The scheduler itself lives on the free server and pulls one task per run with a cron line:

# every quarter hour on the free server
*/15 * * * * cd /opt/summarizer && python run.py --tier cheap

Keep a tiny key-value store of task IDs with their latest status. A finished task gets a marker that makes the next run skip the work, and a failed task keeps an attempt count so the backoff calculation uses real data instead of guesswork. That idempotency is what lets you replay a queue safely after you deploy the fix.

Prove the fix with a poisoned fixture instead of trusting a code review. Drop a malformed message into the queue, run the scheduler, and assert that the runner retries with growing delays and then marks the task dead rather than hammering the provider. That one assertion converts the postmortem from a story into a regression suite.

What this approach cannot do

Be honest about the boundaries before you adopt the pattern. Regulated or customer-identifiable data should not pass through this setup unless your own compliance review has explicitly cleared the endpoint, and latency-critical user-facing calls have no place on a free server. If your workload needs consistent throughput at peak hours, provision for that with a real contract instead of an experimental host.

The teams that benefit most run background autonomy: nightly summarizers, triage agents, and internal tooling where a response in five minutes is perfectly acceptable. Teams with strict data residency rules and agents that take irreversible actions should keep the routing idea but leave the free server out of the architecture entirely.

If you have lived through one of these nights, the cheapest win is not a bigger account; it is a budget guard, a tier router, and a retry policy that knows when to stop. Pick one boring job, move it to the free tier this week, keep the frontier model for the task that truly needs it, and write the postmortem before the next incident writes it for you.