Everyone tracks the cost per token. It's the obvious metric. But if that's all you're watching, you're missing the bigger picture. After spending way too much time sifting through invoices and logs, I've found the real cost sinks are often hidden elsewhere.
1. The Retries & Timeouts Tax
Your code retries on a 503 from OpenAI. Standard practice, right? But are you tracking the cost of those retries? A temporary outage or a poorly optimized prompt can cause a spike in retries, doubling or tripling the cost of a single user action without you even noticing until the end of the month. It's not just the API cost, either. It's the extended function execution time, the user waiting, the potential for cascading failures.
2. The "Which User Was That?" Problem
A huge bill comes in. You see a spike in gpt-4-turbo usage last Tuesday. Who caused it? Was it a single power user, a misbehaving script, or a feature getting abused? If you're just passing an API key from your backend, you have no idea. Per-user attribution isn't a vanity metric; it's essential. Without it, you can't tell who your most expensive users are, or if a specific user is hammering your service in a way you didn't anticipate.
3. The "Development vs. Production" Blind Spot
You run tests, you experiment in a staging environment. All those calls are hitting your single API key. How much are you spending on non-production traffic? Is your CI/CD pipeline making a dozen LLM calls on every commit? These small, untracked costs from multiple developers and automated systems add up. They muddy the waters, making it impossible to see your actual production COGS.
I built LLMeter to get a handle on this. It's an open-source dashboard that helps me see costs per user, per model, and set alerts before the bill gets out of hand. It directly connects to OpenAI, Anthropic, and others, giving me a real-time view of what's actually happening. Fwiw, it's helped me catch more than one runaway script. Check it out if you're tired of flying blind: https://llmeter.org
Top comments (0)