The Hidden Costs of AI: Actually Benchmarking Your LLM Calls

You shipped an AI feature. It works. Users love it. Then your cloud bill shows up and suddenly you're rethinking life choices.

The Problem Nobody Talks About

Everyone benchmarks latency. Nobody benchmarks cost-per-call until it's too late.

I spent a week optimizing prompt structure and model selection for one of our features. Dropped response time by 200ms. Looked great in the demo. Then I actually ran the math on token costs and realized I'd optimized for the wrong thing entirely.

Here's what actually matters:

• Cost per request (tokens in + tokens out × model pricing)
• Cache hit rate (if using prompt caching)
• Failure rate (retry costs compound fast)
• User value per call (is this call worth 0.001 cents?)

Actually Measuring This

Most people log something like:

const response = await openai.chat.completions.create({ ... });
console.log(response.usage.total_tokens);

That's fine. That gets you nowhere useful.

What you need:

const call = {
  model: 'gpt-4o-mini',
  input_tokens: response.usage.prompt_tokens,
  output_tokens: response.usage.completion_tokens,
  cache_creation_tokens: response.usage.cache_creation_input_tokens || 0,
  cache_read_tokens: response.usage.cache_read_input_tokens || 0,
  timestamp: new Date(),
};

// Cost math for GPT-4o mini:
// Input: $0.00015 / 1K tokens
// Output: $0.0006 / 1K tokens
// Cache write: $0.00015 / 1K tokens (one-time)
// Cache read: $0.000015 / 1K tokens (90% cheaper!)

const inputCost = (call.input_tokens - call.cache_write_tokens) * 0.00000015;
const outputCost = call.output_tokens * 0.0000006;
const cacheWriteCost = call.cache_write_tokens * 0.00000015;
const cacheReadCost = call.cache_read_tokens * 0.000000015;

const totalCost = inputCost + outputCost + cacheWriteCost + cacheReadCost;

// Log this somewhere useful (database, analytics, whatever)
db.llm_calls.insert({
  ...call,
  cost_cents: totalCost * 100,
});

Real-World Impact

We had a feature that generated personalized reports. Each report = 3-4 API calls. ~500 users/day.

Before benchmarking: Thought it was working fine, ~0.15 per report.

After actually measuring:

• 12% of calls were retries (duplicates, cost × 1.5)
• Cache was never warmed (easy win, 90% cheaper reads)
• Reports were using GPT-4 when GPT-4o mini was 70% cheaper

After fixing: $0.04 per report. That's a 73% reduction. For 500 users/day, that's $1,500/month back in our pocket.

The Checklist

Before you ship AI features:

• [ ] Log every call with input/output token counts
• [ ] Categorize by feature/endpoint (so you can see where money bleeds)
• [ ] Calculate real costs using current pricing
• [ ] Set alerts if cost-per-request spikes
• [ ] Review monthly — compare to predictions
• [ ] If using cached prompts, actually measure cache hit rate
• [ ] Test model downgrades (mini vs base vs pro) for quality delta

The tricky part: quality matters. You can't just use the cheapest model. But you probably can use a cheaper model than you think.

Tools That Help

• Opentelemetry + OpenAI integration — structured logging for all calls
• Langsmith — tracks costs by chain, shows you bottlenecks
• Custom logging layer — honestly, just a database table. You don't need much.
• Cost alerts — Datadog/New Relic will scream if costs spike

The Honest Take

This isn't sexy work. It won't make your demo faster. It won't impress anyone.

But it's the difference between a profitable AI feature and one that hemorrhages money in production. I'd rather spend a day measuring than lose $1,000/month wondering where it went.

---

Want to stay sharp on AI tools and strategies that actually save money? Check out the LearnAI Weekly newsletter — real benchmarks, honest takes, no fluff.