Teams that ship AI features usually track latency and error rates, then discover at month-end that the bill tripled. Token costs are observable too — if you instrument the right metrics per endpoint.
The four metrics that matter
Tokens in / tokens out per endpoint. Log prompt size and completion size on every call. The endpoints with the longest prompts are almost always your most expensive ones.
Effective rate per 1M tokens. Your gateway or provider's list price is not your effective rate once caching, retries and model mix are factored in. Track actual spend divided by actual tokens.
Cache hit ratio. If your workload re-uses long system prompts, cache reads can cut input cost dramatically. If you don't measure hit ratio, you can't negotiate or route for it.
Cost per successful user action. Tokens per request is an engineering metric. Cost per signed-up user or resolved ticket is a business metric — and the one your roadmap decisions should depend on.
Making comparisons honest
When you compare OpenAI API pricing across providers or gateways, normalize to cost per 1M input and output tokens, and check whether cache and batch discounts are actually passed through.
We publish our full per-token table — the KeyoAPI OpenAI rates — on one page: KeyoAPI. Same OpenAI-compatible SDK, change only the base URL.
Instrumentation in an afternoon
Wrap your LLM client once: log tokens, model, latency and cost per call. Export to your existing metrics stack. Within a week you'll know which endpoint to optimize first — and whether a routing change actually saved money.
Top comments (0)