How to Prevent Runaway LLM API Bills: Building an In-Process Circuit Breaker in Python & TypeScript

#webdev #programming #ai #python

Every developer building autonomous agents, RAG pipelines, or LLM-powered backends shares the same quiet fear: waking up to an unexpected $300+ bill from OpenAI or Anthropic because of a runaway while-loop.

Here is how that scenario usually happens:

You deploy an autonomous agent with tool-calling capabilities.
A tool returns an unexpected JSON schema or an unhandled edge-case exception.
The agent enters a retry loop with a 4,000+ token context window.
It fires 50 times in 20 minutes before failing completely.

The worst part? Cloud provider billing alerts (OpenAI, Anthropic, AWS) only send notification emails hours after the budget has already been exhausted.

In this post, we'll look at how to implement an in-process local circuit breaker with zero external dependencies to prevent this from ever happening.

The Two Pillars of LLM Safety

To protect your API keys and credit cards locally, your application needs two safety mechanisms:

Hard USD Budget Caps: An in-memory ledger that calculates token cost in real time and throws an immediate BudgetExceededError if an operation would exceed your chosen threshold (e.g. $1.00 per session).
Cyclic Loop Detection: A rolling action-signature hash that identifies when an agent is repeating the exact same tool call or prompt sequence in a loop.

1. Implementing Cyclic Loop Detection (Python)

When agents get stuck, they almost always repeat identical or near-identical tool signatures. We can detect this with a lightweight SHA-256 rolling window:

import hashlib

class LoopBreaker:
    def __init__(self, max_repeat_actions: int = 3):
        self.max_repeat_actions = max_repeat_actions
        self.action_hashes = []

    def check(self, action_signature: str) -> None:
        """
        Hashes action signatures to catch repetitive agent behavior.
        Throws a RuntimeError if the same action repeats N consecutive times.
        """
        h = hashlib.sha256(action_signature.encode("utf-8")).hexdigest()
        self.action_hashes.append(h)

        # Inspect the most recent actions
        recent = self.action_hashes[-self.max_repeat_actions :]
        if len(recent) >= self.max_repeat_actions and len(set(recent)) == 1:
            raise RuntimeError(
                f"[CircuitBreaker] Infinite loop detected! The action '{action_signature[:60]}...' "
                f"was repeated {self.max_repeat_actions} consecutive times. Tripping circuit."
            )

Now, inside your agent execution loop:

breaker = LoopBreaker(max_repeat_actions=3)

while agent.is_running:
    tool_call = agent.get_next_action()

    # Check signature before executing
    breaker.check(f"{tool_call.name}:{tool_call.args}")

    result = tool_call.execute()

2. Real-Time Token Cost Calculation

Instead of waiting for an invoice, we calculate the cost directly from response.usage tokens using current per-million token rates:

Total Cost = (Input Tokens ÷ 1,000,000 × Input Rate) + (Output Tokens ÷ 1,000,000 × Output Rate)

class BudgetGuard:
    def __init__(self, max_budget_usd: float = 2.00):
        self.max_budget_usd = max_budget_usd
        self.total_spent_usd = 0.0

    def record(self, input_tokens: int, output_tokens: int, input_rate_1m: float, output_rate_1m: float):
        cost = (input_tokens / 1_000_000) * input_rate_1m + (output_tokens / 1_000_000) * output_rate_1m

        if self.total_spent_usd + cost > self.max_budget_usd:
            raise Exception(f"Budget exceeded! Attempted spend would reach ${self.total_spent_usd + cost:.4f} (Cap: ${self.max_budget_usd:.2f})")

        self.total_spent_usd += cost
        return cost

Live Interactive Cost Calculator

To help developers estimate their API spend before building, I built a free, zero-dependency web calculator comparing live token costs across 13+ frontier models (OpenAI , Claude Opus , Gemini ):

👉 Try the Live LLM Cost Calculator

The entire project is open-source on GitHub:

⭐ GitHub: manoharpiridi-stack/llm-cost-calculator

Production-Ready Package: LLM Token Guard

If you want the complete, pre-built production package containing:

Pre-configured token pricing tables for 15+ models across OpenAI, Anthropic, and Google
Both pure Python and TypeScript / Next.js implementations
Slack/Discord webhook alerts on budget breach
100% passing unit test suite and commercial license

You can grab the turnkey package here:

🛡️ Get LLM Token Guard on Polar ($19)

Conclusion

Whether you build your own circuit breaker or use a pre-built one, never deploy an autonomous agent loop without in-process budget stops. Cloud provider billing alerts are designed for accounting, not real-time runtime safety.

What strategies do you currently use to monitor your LLM token spend in production? Let's discuss in the comments below!

DEV Community