Postmortem: The Prompt Template That Silently Upgraded Every Request

Some incidents do not announce themselves with an outage; they arrive as a number that looks wrong on a cost dashboard at 2 a.m. The summarization pipeline in this postmortem kept serving every single request, which made the alert easy to dismiss, but the queue was growing and the spend chart was climbing like a hockey stick. A service that works perfectly while it burns money is still an incident, and the money is usually the first honest signal you get.

This story belongs to a class of failures you have probably touched before: AI-assisted review pipelines that were supposed to make teams faster. The irony is that nobody reviewed the reviewer itself, and a single prompt template change was enough to silently upgrade every request to the most expensive model available. You can reproduce the whole incident in about twenty minutes with the router code below, because the details are constructed rather than remembered; the failure pattern, unfortunately, is very real.

The Pipeline

The design was unremarkable, which is exactly why it survived review. Support threads arrived, a small classifier scored how painful each one would be to summarize, and a router sent the easy traffic to a free lane while the hard traffic went to a paid frontier model. The classifier was prompt-based, which meant someone could improve its behavior by editing a template without touching application code, and that convenience became the root cause of everything that followed.

The traffic mix stayed healthy for months, and the cost dashboard was so boring that nobody looked at it. Then a developer trimmed the classifier prompt to save tokens, because the old wording contained phrases like 'escalation', 'refund', and 'account locked' that felt redundant. The new template removed those trigger words in the name of efficiency, and the classifier responded by returning low-confidence scores for nearly every thread.

Timeline

The timeline below comes from a constructed replay rather than a production incident, and the shape of the curve is the point. Each step was sensible in the moment, and that is what makes this failure so easy to repeat.

14:02  Deploy prompt template v3; classifier context trimmed.
14:47  Cost alert: hourly spend passes 2.1x the rolling baseline.
15:30  Queue depth climbs; workers start retrying timed-out requests.
16:12  Log query shows 91% of traffic took the fallback path.
16:58  Rollback of template v3; spend returns to baseline.

The router treated low confidence as a reason to escalate to the paid lane, because the old classifier rarely said "I don't know" and the fallback was written for that edge case. The retry loop made things worse, because every timed-out request was re-scored and routed straight back to the expensive model instead of being parked for a human. Three design decisions turned a small regression into an expensive one: the fallback chose the paid model silently, the alert watched absolute spend instead of lane mix, and the worker retried failures by re-running the router rather than failing into a dead-letter queue. Each choice looked reasonable in isolation, and together they created a system where the worst case was also the most expensive possible outcome.

The Durable Fix

The durable fix is to invert the default: the free lane is the resting state, and the paid lane requires a positive signal with a logged reason. Low confidence now means 'send it to the free lane', and the paid lane is reserved for threads that are explicitly sensitive or genuinely hard:

def route(prompt, score):
    if score < 0.6:
        return {'lane': 'free', 'reason': 'low_confidence', 'retry': False}
    if is_sensitive(prompt):
        return {'lane': 'paid', 'reason': 'pii_or_legal', 'retry': True}
    return {'lane': 'free', 'reason': 'default', 'retry': True}

The retry path carries the same discipline, because a free-lane failure should never silently upgrade itself into a paid-lane invoice:

def worker(prompt, max_retries=3):
    for attempt in range(max_retries):
        lane = route(prompt, classify(prompt))
        try:
            return call(lane)
        except RateLimitError:
            if attempt == max_retries - 1:
                return dead_letter(prompt)  # humans, not a bigger bill
            time.sleep(2 ** attempt)

For the free lane itself the design needed a provider whose access could absorb routine traffic without a credit card, and MonkeyCode's free model access filled that role for this workload. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The same project also offers a free server option, which became the home for the batch worker, so the orchestration layer stopped burning paid compute minutes while it waited on model calls. The free allowance was advertised as ten million tokens when this was written, but free quotas change with product decisions, so verify the current numbers in the project's official page before you build anything on them.

The second half of the fix is monitoring the mix rather than the total. A lane-mix metric that alarms when the paid lane exceeds ten percent of traffic catches the next template change in minutes, while an absolute spend alert can take hours to mean anything. Alert on that ratio, and the deployment that caused this incident gets rolled back before the queue even notices.

How to Recognize It Early

The fastest way to spot this incident in your own system is to ask one boring question: what does the fallback path do? Run the diagnostic below against a day of router logs, and if the fallback lane is a paid model, you have the bug even when the numbers look calm.

python - <<'EOF'
import collections, json
lanes = collections.Counter(json.loads(line)['lane'] for line in open('router.log'))
paid = lanes['paid'] / sum(lanes.values())
print('paid-lane share: {:.0%}'.format(paid))
EOF

A paid-lane share above ten percent is not automatically wrong, but it should be a conscious decision with a named owner attached to it, not a silent default. Free quotas also shift, so keep a configuration layer that lets you swap the lane provider without rewriting the router.

Limitations

This approach is not a universal recommendation, and the free lane earns its name with real constraints. Rate limits and latency variance are the price of the free server, and if your workload needs guaranteed response times or handles regulated data, the paid lane is the correct home for those requests. If your team cannot tolerate a batch job finishing an hour late, or if your security review has not cleared a free server for your data, this architecture is not for you; the free lane is a cost optimization for forgiving traffic, not an SLA for your customers.

The lesson from this postmortem is that the cheapest lane only stays cheap when failures route traffic away from it, never toward it. If you want to experiment with a free lane that can absorb real volume, MonkeyCode is a reasonable place to start, but treat its free tier as a constraint to engineer around rather than a promise to depend on. The code above gives you everything you need to break your own router in a sandbox before production does it for you.