My LLM Bill Kept Growing, but User Traffic Didn’t

My request count looked normal.

Traffic was mostly flat. There was no sudden wave of new users, no runaway background job, and no obvious model upgrade hiding in a deployment.

But the LLM bill kept climbing.

My first instinct was to look for the usual suspects: a pricing change, an expensive model accidentally pushed to production, or someone hammering an endpoint.

None of those explanations held up.

The problem was simpler: I was measuring requests, while the provider was charging me for tokens.

Those two graphs had slowly stopped telling the same story.

A request is not a useful cost unit

I had dashboards for request volume, latency, error rate, and model name. That felt like enough observability until cost started behaving differently from traffic.

A single LLM request can contain:

• the system prompt
• the current user message
• conversation history
• retrieved documents
• tool definitions
• JSON schemas
• examples included in the prompt
• output tokens
• provider-specific reasoning tokens
• another attempt after a retry

Two requests to the same endpoint can therefore have completely different costs.

One might classify a short support message. Another might resend a long conversation, twelve tool definitions, and several retrieved documents before producing a two-sentence answer.

My dashboard counted both as one request.

The invoice did not.

The first number I needed was tokens per operation

Logging total tokens was better than logging nothing, but it still left me with a pile of numbers and no explanation.

I needed to connect usage to the work my application was trying to perform.

For each model call, I started recording:

• operation_id: the user-visible action
• workflow: chat, summarization, extraction, agent step, and so on
• model
• attempt
• input_tokens
• output_tokens
• cached_input_tokens, when reported
• reasoning_tokens, when reported
• latency_ms
• status
• whether the call was a retry or fallback

The distinction between a request and an operation matters.

If a user clicks “Generate” once and the application makes three model calls, I want to see one operation with three attempts or steps—not three unrelated requests.

That was the first change that made the cost graph useful.

A small usage logger

Here is a simplified Node.js example using an OpenAI-compatible Chat Completions endpoint.

The detailed usage fields are optional because providers and models do not all return the same response shape.

import crypto from "node:crypto";

const baseUrl = process.env.LLM_BASE_URL ?? "https://api.openai.com/v1";
const apiKey = process.env.LLM_API_KEY;
const model = process.env.LLM_MODEL;

if (!apiKey || !model) {
  throw new Error("Set LLM_API_KEY and LLM_MODEL");
}

function readUsage(usage = {}) {
  return {
    inputTokens: usage.prompt_tokens ?? 0,
    outputTokens: usage.completion_tokens ?? 0,
    totalTokens: usage.total_tokens ?? 0,
    cachedInputTokens:
      usage.prompt_tokens_details?.cached_tokens ?? 0,
    reasoningTokens:
      usage.completion_tokens_details?.reasoning_tokens ?? 0,
  };
}

async function callModel({
  messages,
  workflow,
  operationId = crypto.randomUUID(),
  attempt = 1,
}) {
  const startedAt = Date.now();

  const response = await fetch(`${baseUrl}/chat/completions`, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      Authorization: `Bearer ${apiKey}`,
    },
    body: JSON.stringify({
      model,
      messages,
    }),
  });

  const body = await response.json();
  const usage = readUsage(body.usage);

  const event = {
    timestamp: new Date().toISOString(),
    operationId,
    workflow,
    model,
    attempt,
    status: response.ok ? "success" : "error",
    latencyMs: Date.now() - startedAt,
    ...usage,
  };

  console.log(JSON.stringify(event));

  if (!response.ok) {
    throw new Error(
      body.error?.message ?? `LLM request failed: ${response.status}`
    );
  }

  return {
    operationId,
    text: body.choices?.[0]?.message?.content ?? "",
    usage,
  };
}

const result = await callModel({
  workflow: "support_reply",
  messages: [
    {
      role: "system",
      content: "Write concise and accurate support replies.",
    },
    {
      role: "user",
      content: "How do I update my billing email?",
    },
  ],
});

console.log(result.text);

In production, I send this event to the same logging or tracing system that holds the rest of the workflow.

The important part is not the destination. It is preserving the relationship between usage and the operation that caused it.

Where the extra tokens were hiding

Once I grouped usage by workflow and operation, several patterns became visible.

Conversation history kept growing

The chat endpoint received the full conversation history on every turn.

A conversation that began cheaply became progressively more expensive even though the user was still sending short messages.

Request volume stayed flat. Input tokens per request did not.

The fix was not simply “truncate everything.” Older context sometimes mattered.

Instead, I separated context into three groups:

• recent messages that should be preserved
• durable facts that could be stored separately
• older conversation that could be summarized or dropped

The important measurement was input tokens by conversation age. Without that breakdown, long-running chats looked like ordinary traffic.

Tool definitions were being sent repeatedly

My agent had access to several tools, each with a description and JSON schema.

Those definitions were part of the model input. I had been thinking about them as application configuration, not recurring context.

Some workflows exposed tools they could never use. Others included verbose parameter descriptions copied from internal documentation.

Reducing the tool set per workflow made the agent easier to reason about and reduced repeated input.

It also gave me a better question than “How do I shorten this prompt?”

The better question was:

Why am I sending this piece of context on this specific call?

Retries looked like new work

A timeout triggered a retry. A parsing failure triggered another retry. A fallback model sometimes ran after both.

At the infrastructure level, those were separate requests.

At the product level, the user had asked for one thing.

Without an operation ID and attempt number, I could not distinguish growing usage from growing demand.

I now track both:

requests per operation
tokens per operation

If requests per operation rises, I look at retries, fallbacks, agent loops, and validation failures before blaming user traffic.

I also avoid assuming that every failed request has zero cost. Whether usage is recorded depends on the provider and how far the request progressed, so application logs should be reconciled with provider-side usage data.

Outputs were longer than the product needed

Some workflows asked for structured data containing five fields but allowed the model to generate a long explanation around it.

Another workflow produced detailed internal reasoning-style text that the UI never displayed.

The model was doing more work than the product used.

Output limits helped, but clearer response contracts helped more. If the UI needs a category, confidence score, and short explanation, the prompt and schema should request exactly that.

“Be concise” is weaker than defining the output the application can actually consume.

The dashboard I use now

I no longer begin cost investigations with total spend.

I start with four ratios:

input tokens / operation
output tokens / operation
model calls / operation
cached input tokens / input tokens

Then I break them down by:

• workflow
• model
• release version
• customer or plan, using an internal non-sensitive identifier
• success, retry, and fallback status

This makes different problems look different.

If input tokens per operation rise, context may be growing.

If calls per operation rise, retries or loops may be involved.

If output tokens rise, the response contract may have drifted.

If the cache ratio changes, a supposedly stable prompt prefix may no longer be stable. Prompt caching behavior and usage reporting vary by provider and model, so I treat the returned usage fields and provider documentation as the source of truth.

I stopped estimating cost from averages

A global “average cost per request” hid too much.

A cheap classification call and a long agent workflow should not share the same baseline. Neither should successful operations and operations that needed three attempts.

I now compare each workflow against its own recent behavior.

The alert is not:

LLM spend increased by 20%

It is closer to:

input tokens per successful support_reply operation
increased materially after the latest release

That points toward something I can investigate.

A total-spend alert tells me there is a fire. A workflow-level usage alert tells me which room might be burning.

Cost optimization came later

Before this work, my optimization ideas were mostly guesses:

• use a cheaper model
• shorten the system prompt
• reduce output length
• add caching
• retrieve fewer documents

All of those can be reasonable. None of them tells you where your application is wasting tokens.

Once usage was tied to operations, the priorities changed.

The biggest opportunity was not necessarily switching models. Sometimes it was stopping an unnecessary second call. Sometimes it was removing irrelevant tools. Sometimes it was preventing the same context from growing forever.

The useful sequence became:

1. Measure usage per operation.
2. Find the workflow whose token intensity changed.
3. Separate input, output, retries, fallbacks, and caching.
4. Change one behavior.
5. Confirm that quality and reliability did not regress.

Only then did model pricing become a meaningful optimization variable.

The lesson

User traffic and LLM usage are related, but they are not the same metric.

Traffic tells me how many people are using the product.

Token telemetry tells me how much work the application is asking the models to do on their behalf.

When those numbers diverge, I no longer start by blaming the model or the provider. I look at context growth, tool payloads, retries, fallbacks, and workflow design.

The bill was not the first signal.

It was just the first signal I had bothered to watch.

I work on TokenBay, so multi-model usage and routing are problems I spend a lot of time thinking about. The most useful improvement, though, was provider-independent: attach every token to a workflow and an operation before trying to optimize it.