I Built a $400/mo LLM Cost Monitoring System (Here's What I Learned)

#ai #llm #devops

Six months ago I got a $3,000 LLM bill. I had no idea where it came from. Now I have a monitoring system that tracks every call. Here's what I built.

What Happened

I was running a small SaaS with GPT-4. The bill came in at $3,127 for the month. I had maybe 500 paying users. That's $6/user in LLM costs. My product was $20/month. I was losing money on every power user.

I had no idea what was happening because I wasn't tracking:

Cost per user
Cost per feature
Cost per model
Request volume

The System I Built

1. Per-Call Logging

def llm_call(messages, model="gpt-4o"):
    start = time.time()
    response = openai.ChatCompletion.create(
        model=model,
        messages=messages
    )

    # Log everything
    cost = calculate_cost(model, response.usage)
    log({
        "model": model,
        "prompt_tokens": response.usage.prompt_tokens,
        "completion_tokens": response.usage.completion_tokens,
        "cost": cost,
        "latency_ms": (time.time() - start) * 1000,
        "user_id": get_current_user(),
        "feature": get_current_feature()
    })
    return response

2. Cost Dashboard

I built a simple dashboard that shows:

Daily/weekly/monthly cost
Cost by model
Cost by user
Cost by feature
Projected month-end cost

3. Alerting

if daily_cost > DAILY_BUDGET * 0.8:
    send_alert(f"80% of daily budget used: ${daily_cost}")

if monthly_cost > MONTHLY_BUDGET * 0.9:
    send_alert(f"90% of monthly budget used: ${monthly_cost}")

What I Found

After a month of monitoring:

60% of costs came from one "help me write emails" feature
20% came from a chatbot feature I thought was cheap
Users who used the email feature used 10x more tokens than average

I killed the email feature. Costs dropped 60%. Users complained. I didn't care.

The Numbers

Month	LLM Cost	Users	Cost/User
January	$3,127	500	$6.25
February	$1,100	520	$2.12
March	$890	540	$1.65

The monitoring paid for itself in one month.

What I'd Do Differently

Start with cost monitoring, not features. Every LLM feature should have a cost model.
Set budgets before you build. Know your acceptable cost per user before adding LLM features.
Monitor per-feature, not just total. Aggregate monitoring hides which feature is killing you.