Why Your AI Agents Are Burning Cash and How to Fix It

If you've deployed LLM-powered agents in production, you've probably watched your API bill climb while your agents keep making the same dumb mistakes. I hit this wall about two months ago — an agent pipeline that cost $40/day to run and still couldn't reliably handle edge cases it had already seen.

The frustrating part? The agents weren't getting better over time. Every request started from scratch, burning through tokens on problems they'd already solved.

Let me walk through what's actually going wrong and how open-source tooling like OpenSpace can help you build agents that learn, cost less, and genuinely improve over time.

The Root Cause: Stateless Agents Are Expensive Agents

Most agent frameworks treat every interaction as a blank slate. Your agent receives a task, spins up a chain of LLM calls with full system prompts, tool descriptions, and few-shot examples — then throws all that context away when it's done.

This creates three compounding problems:

• Redundant token usage — the same lengthy prompts get sent hundreds of times a day
• No learning loop — mistakes don't inform future behavior
• Prompt bloat — developers keep adding instructions to handle edge cases, making every call more expensive

Here's what a typical naive agent loop looks like:

# The expensive way — full context on every single call
def handle_task(task: str):
    messages = [
        {"role": "system", "content": MASSIVE_SYSTEM_PROMPT},  # 2000+ tokens every time
        {"role": "user", "content": task}
    ]
    # Each call includes ALL tool definitions, ALL examples
    response = llm.chat(messages, tools=ALL_TOOLS)  # $$$

    while response.needs_action:
        # Recursive calls with growing context windows
        messages.append(response.message)
        messages.append(execute_tool(response.tool_call))
        response = llm.chat(messages, tools=ALL_TOOLS)  # More $$$

    return response.content
    # Context is gone. Lessons learned? Zero.

Every call carries the full weight of your system prompt and tool definitions. If your agent takes 5 steps to complete a task, you're paying for that system prompt 5 times.

Step 1: Implement Experience-Based Prompt Optimization

The first fix is to stop treating prompts as static artifacts. OpenSpace — an open-source framework from HKUDS — approaches this by letting agents evolve their own prompts based on what actually works.

The idea is straightforward: track which prompt patterns lead to successful outcomes, then automatically refine the prompts over time.

from openspace import AgentSpace

# Initialize with your base agent configuration
agent_space = AgentSpace(
    model="gpt-4o-mini",  # Start with a cheaper model
    optimization_target="cost_and_accuracy"
)

# Register your agent's task handler
@agent_space.task("data_extraction")
def extract_data(input_doc):
    # OpenSpace tracks success/failure of each run
    # and uses that signal to optimize the prompt
    result = agent_space.run(
        task_input=input_doc,
        eval_fn=lambda r: validate_extraction(r)  # Your quality check
    )
    return result

# After N runs, trigger self-optimization
agent_space.optimize()  # Refines prompts based on accumulated experience

The key insight here: instead of you manually tweaking prompts when something breaks, the framework collects execution traces and figures out what phrasing, structure, and examples actually produce good results.

Step 2: Cut Costs with Adaptive Model Selection

Not every agent call needs GPT-4. That's obvious in theory, but hard to implement in practice because you don't always know which calls are simple and which are complex until after the fact.

The self-evolving approach solves this by learning which task patterns can be safely routed to cheaper models:

# Configure a model cascade — try cheap first, escalate if needed
agent_config = {
    "models": [
        {"name": "gpt-4o-mini", "cost_per_1k": 0.00015, "priority": 1},
        {"name": "gpt-4o", "cost_per_1k": 0.005, "priority": 2},
    ],
    "routing": {
        "strategy": "confidence_based",
        "confidence_threshold": 0.85,  # Only escalate when uncertain
        "learn_from_history": True      # Gets smarter about routing over time
    }
}

# Over time, the router learns:
# - Simple lookups → always use mini (saves 97% per call)
# - Complex reasoning → route to gpt-4o
# - Ambiguous cases → try mini first, escalate if confidence is low

I've seen this pattern alone cut costs by 60-70% on pipelines that were previously using a single expensive model for everything. The trick is that the routing itself improves as the system accumulates more execution data.

Step 3: Build a Feedback Loop That Actually Works

The "self-evolving" piece is what makes this approach stick long-term. Without it, you're just doing a one-time optimization that degrades as your data changes.

Here's how to wire up a proper feedback loop:

# Capture execution traces with outcome signals
class AgentTracer:
    def __init__(self):
        self.traces = []

    def record(self, task_input, steps, outcome, cost):
        self.traces.append({
            "input": task_input,
            "steps": steps,           # What the agent actually did
            "outcome": outcome,       # success/failure + quality score
            "cost": cost,             # Total tokens and dollars
            "timestamp": time.time()
        })

    def get_improvement_signals(self):
        # Find patterns: which step sequences lead to failures?
        # Which prompts produce the best cost/quality ratio?
        failures = [t for t in self.traces if not t["outcome"]["success"]]
        expensive_wins = sorted(
            [t for t in self.traces if t["outcome"]["success"]],
            key=lambda t: t["cost"], 
            reverse=True
        )
        return {
            "failure_patterns": self._cluster(failures),
            "optimization_candidates": expensive_wins[:10]  # Costly but correct
        }

The valuable insight is in optimization_candidates — these are tasks the agent got right but spent too much to solve. They're your best targets for cost reduction because you already know the correct answer to optimize toward.

Prevention: Stop the Bleeding Before It Starts

Before you refactor your entire agent stack, here are some quick wins:

• Set budget caps per agent call — if a single task exceeds $0.50 in API costs, kill it and log the trace for review
• Cache deterministic tool calls — if your agent calls the same API with the same params, cache it (you'd be amazed how often this happens)
• Trim your system prompts — most system prompts I've audited contain 30-40% redundant or contradictory instructions
• Log everything — you can't optimize what you can't measure; track tokens, latency, and outcome quality per task type

The Bigger Picture

What projects like OpenSpace are pushing toward is a fundamental shift in how we think about agents. Instead of static prompt-and-pray pipelines, we're moving toward systems that treat agent behavior as something that can be empirically optimized — more like training a model than writing a script.

The practical takeaway: if your agents aren't getting cheaper and better over time, you're leaving money on the table. Start with execution tracing, add outcome evaluation, then layer in automated optimization. You don't need to adopt an entire framework on day one — even basic feedback loops will show results within a few hundred runs.

I'm still experimenting with different optimization strategies on my own projects, and I haven't tested every feature thoroughly yet. But the core pattern — track outcomes, learn from them, evolve the agent — is sound engineering regardless of which specific tools you use to implement it.