Your AI Chatbot Has Freeloaders: A Developer's Guide to Killing Runaway Token Bills

You shipped a chatbot. Users love it. Then the invoice lands and finance asks why your "cheap AI feature" costs more than a junior engineer's salary.

Welcome to the token economy, where a single verbose system prompt multiplied across 40,000 daily requests turns into a five-figure surprise. The good news: most runaway LLM bills come from a handful of fixable patterns. Let's find them.

Where the money actually leaks

Before optimizing anything, understand what you're paying for. Every LLM call bills on two axes: input tokens (your prompt, history, retrieved context) and output tokens (the model's reply). Output usually costs 3-5x more than input.

The usual suspects:

• Bloated system prompts. A 2,000-token instruction block sent on every single request. That's fixed overhead you pay forever.
• Unbounded chat history. Naively appending the full conversation until context windows explode.
• Overkill model selection. Routing "what are your hours?" to GPT-4-class models when a small model would nail it.
• No output caps. Letting the model ramble for 800 tokens when 100 would do.
• Retry storms. Failed calls that silently re-fire and double-bill.

You can't govern what you don't measure. So step one is instrumentation.

Instrument every call

Wrap your LLM client so no request escapes without logging tokens, cost, model, and the feature that triggered it. This one habit surfaces 80% of your waste.

import time
from dataclasses import dataclass

PRICING = {  # USD per 1K tokens
    "gpt-4o":      {"in": 0.005,  "out": 0.015},
    "gpt-4o-mini": {"in": 0.00015,"out": 0.0006},
}

@dataclass
class CallResult:
    text: str
    cost: float

def tracked_completion(client, model, messages, feature, user_id, max_tokens=300):
    start = time.time()
    resp = client.chat.completions.create(
        model=model,
        messages=messages,
        max_tokens=max_tokens,  # hard cap on output spend
    )
    usage = resp.usage
    p = PRICING[model]
    cost = (usage.prompt_tokens/1000)*p["in"] + (usage.completion_tokens/1000)*p["out"]

    log_event({
        "feature": feature,
        "user_id": user_id,
        "model": model,
        "prompt_tokens": usage.prompt_tokens,
        "completion_tokens": usage.completion_tokens,
        "cost_usd": round(cost, 6),
        "latency_ms": int((time.time() - start) * 1000),
    })
    return CallResult(resp.choices[0].message.content, cost)

Now you can slice spend by feature and by user. That second dimension matters more than people expect.

Meet your token freeloaders

Every chatbot has power users, and a small slice of them generates a wildly disproportionate share of cost. Sometimes it's a genuine heavy user. Sometimes it's a script hammering your endpoint. Either way, you need per-user budgets.

A simple rolling budget stops one account from eating your margin:

def enforce_budget(user_id, redis, daily_cap_usd=2.0):
    key = f"spend:{user_id}:{today()}"
    spent = float(redis.get(key) or 0)
    if spent >= daily_cap_usd:
        raise BudgetExceeded(f"{user_id} hit ${daily_cap_usd} cap")
    return spent

def record_spend(user_id, redis, cost):
    key = f"spend:{user_id}:{today()}"
    redis.incrbyfloat(key, cost)
    redis.expire(key, 60 * 60 * 26)  # auto-clean

Call enforce_budget before the request, record_spend after. Freeloaders get a polite "you've hit today's limit" instead of wrecking your P&L.

Route by difficulty, not by default

The single biggest lever is model selection. Most chatbot traffic is trivial - greetings, FAQ hits, simple lookups. Sending all of it to a frontier model is like taking a helicopter to the corner store.

Use a cheap model as your default and escalate only when needed:

A pragmatic routing rule

1. Classify intent with a tiny, fast model (or even regex/embeddings for known FAQs).
2. Serve cached or template answers for high-frequency questions.
3. Escalate to the expensive model only for open-ended reasoning.

A solid RAG setup with cached answers can push 60-70% of traffic away from your priciest model. That alone often halves the bill.

Trim the prompt, cap the output

Two quick wins:

Compress history. Don't send 20 turns of raw chat. Summarize older turns into a short running memory and keep only the last few verbatim. You preserve context at a fraction of the tokens.

Cap output tokens. Set max_tokens aggressively per use case. A support answer rarely needs 500 tokens. If it does, that's a signal your prompt is unfocused.

Also lean on prompt caching where your provider supports it. Static system prompts and repeated context can be cached so you stop paying full price for the same 2,000 tokens on every call.

Set guardrails, then watch them

Cost control isn't a one-time cleanup - it's governance. Bake these into your stack:

• Global daily budget with alerts at 50/80/100%.
• Per-feature and per-user caps so no single path runs away.
• Anomaly alerts on spend-per-request spikes (a sign of a prompt regression or abuse).
• A weekly cost-per-conversation metric. Tie it to the value each conversation produces. If a resolved ticket costs $0.04 in tokens and saves $6 in support time, you're winning. If it costs $3, you're not.

The ROI math that actually matters

A chatbot's ROI isn't "we spent $X on tokens." It's cost-per-outcome versus the alternative. Track cost per resolved ticket, per qualified lead, per completed task - then optimize toward that number, not toward raw token minimization.

Because the goal was never to spend zero. It was to make every token pull its weight.

If your AI spend is climbing faster than the value it returns, that's usually an architecture problem, not a pricing problem - and it's fixable. That's exactly the kind of system design we build at Michael AI: automations and agents with cost governance baked in from day one, so ROI holds up when traffic scales.

---

Originally published at getmichaelai.com