DEV Community

Cover image for LLM Cost Optimization in Python - Token Ledgers, Prompt Caching and Model Routing (82% Cheaper in a Simulated Day)
Mick Michaels
Mick Michaels

Posted on

LLM Cost Optimization in Python - Token Ledgers, Prompt Caching and Model Routing (82% Cheaper in a Simulated Day)

An LLM feature has a cost profile that most backend code does not: every request has a price, and the price depends on text you often do not control. PixelPlex’s breakdown of generative AI economics puts it plainly: APIs lower upfront costs but add ongoing usage fees, and infrastructure and maintenance drive long-term operating costs. The upfront budget gets a spreadsheet. The usage fees get a surprise invoice, because LLM inference cost grows with every user, every chat turn and every token in the prompt.

This post is a code-level take on LLM cost optimization. We build a small Python gateway that every LLM call goes through, then replay one synthetic day of traffic (21,995 requests) through it, adding one optimization at a time and recording the dollar figure after each step:

  1. A per-request ledger: tokens counted with a real tokenizer, dollars from the provider’s price table.
  2. Daily budgets per user and per feature.
  3. Context trimming for chat history.
  4. Provider prompt caching, including a bug that makes caching cost more than no caching.
  5. An exact response cache.
  6. A semantic cache, with a threshold sweep that shows the false hits.
  7. Model routing: cheap model first, escalate when it is not confident.
  8. The Batch API for work that can wait.

The end result on this traffic: $80.37/day → $14.52/day (−81.9%), with answer accuracy going from 99.96% to 99.84%. That last number matters as much as the first.

Setup: Python 3.13, tiktoken 0.14.0, numpy 2.5.4, scipy 1.18.1, scikit-learn 1.9.1, pytest 9.2.0. Every output below is copied from a real run, and 12 pytest tests pin the behavior the post relies on (cache writes, false hits, budget degradation, escalation billing). The LLM itself is a deterministic fake client (no API key, no network), so the numbers are reproducible; the prices it bills come from the provider’s official pricing page.

Prices change. The table below was copied from the OpenAI API pricing page on 2026-10-30. Check the current page before you reuse any dollar figure from this post.

The setup: a gateway, a fake client, a price table

Nothing in the app calls the provider SDK directly. Everything goes through Gateway.handle(), which makes each optimization a config flag that can be switched on and measured.

Request path through the LLM gateway: budget check, exact cache, semantic cache with a length gate, router, provider with prompt cache, ledger

The synthetic day has three features with different cost shapes:

Feature Requests Shape
support_chat 15,695 ~2,000-token policy prompt, multi-turn chats, popular FAQ openers with greetings and typos; one user pastes 15,000-token logs
classify_ticket 6,000 short prompts, 35% are repeats of popular tickets, often in different casing
nightly_digest 300 3,000 to 4,500-word documents, nobody waits for the answer

The fake client returns usage like a real API, simulates prompt-cache reads and writes, and marks each answer correct or wrong based on a hidden task difficulty and a made-up “capability” per model. Those capability numbers are invented for the simulation, so treat the accuracy column as a way to see trade-offs, not as a benchmark.

The two models and their prices (Standard tier, USD per 1M tokens, checked 2026-10-30):

Model Input Cached input Output Role here
gpt-6.1-terra $1.00 $0.05 $5.00 default “strong” model
gpt-6-luna $0.10 $0.01 $0.50 cheap model for routing

For the Batch API the code applies the 50% discount stated in OpenAI’s Batch guide to each Standard price.

Step 1. Count tokens and dollars on every request

LLM token cost is input tokens times the input rate plus output tokens times the output rate, with two twists: cached input is billed at a lower rate, and on newer OpenAI models a cache write is billed at a higher one. The prompt caching guide says that for GPT-5.6 and later, writes cost 1.25× the uncached input rate with the default 30-minute lifetime, and reads cost 0.1× (0.05× for GPT-6.1 Sol and Terra). Each input token is billed at exactly one of these rates. The guide names GPT-6.1 Terra under those rules; applying the same 1.25× write rate to Luna is my assumption, so check your own usage fields.

"""Price table, copied by hand from the provider's official pricing page.

Prices change. Re-check them before you trust any dollar figure below.
"""
from dataclasses import dataclass

PRICES_CHECKED_ON = "2026-10-30"
PRICES_SOURCE = "OpenAI API docs: Pricing (Standard and Batch tables)"


@dataclass(frozen=True)
class Price:
    input: float         # USD per 1M uncached input tokens
    cached_input: float  # USD per 1M input tokens served from the prompt cache
    output: float        # USD per 1M output tokens

    @property
    def cache_write(self) -> float:  # 30-minute cache writes: 1.25x uncached input
        return self.input * 1.25


# Standard tier, USD per 1M tokens
STANDARD = {
    "gpt-6.1-terra": Price(input=1.00, cached_input=0.05, output=5.00),
    "gpt-6-luna": Price(input=0.10, cached_input=0.01, output=0.50),
}
# Batch API: 50% off both input and output, results within 24 hours
BATCH = {m: Price(p.input / 2, p.cached_input / 2, p.output / 2) for m, p in STANDARD.items()}


@dataclass(frozen=True)
class Usage:
    input_tokens: int       # total prompt tokens, cached ones included
    cached_tokens: int = 0  # part of input_tokens read from the prompt cache
    cache_write_tokens: int = 0  # part of input_tokens written to the cache
    output_tokens: int = 0


def cost_usd(model: str, usage: Usage, batch: bool = False) -> float:
    p = (BATCH if batch else STANDARD)[model]
    uncached = usage.input_tokens - usage.cached_tokens - usage.cache_write_tokens
    return (
        uncached * p.input
        + usage.cached_tokens * p.cached_input
        + usage.cache_write_tokens * p.cache_write
        + usage.output_tokens * p.output
    ) / 1_000_000
Enter fullscreen mode Exit fullscreen mode

Before a call, you need an estimate so you can check the budget. tiktoken is OpenAI’s BPE tokenizer, and in v0.14.0 its model table maps GPT-4o, GPT-4.1, GPT-5 and the o-series to the o200k_base encoding. The provider’s usage object is still the billing truth: the models page does not say which encoding the GPT-6 family uses, and per-message overhead is approximate. For Claude, use Anthropic’s free /v1/messages/count_tokens endpoint instead, which its docs also describe as an estimate.

"""Pre-call token estimates with tiktoken.

The provider's `usage` object is the billing truth. This estimate exists so you
can check a budget *before* you spend money, and so the fake client in this
repo produces token counts that look like real ones.
"""
from functools import lru_cache

import tiktoken

ENCODING = "o200k_base"
PER_MESSAGE_OVERHEAD = 4  # role + separators; an approximation, reconcile with usage


@lru_cache(maxsize=1)
def _enc():
    return tiktoken.get_encoding(ENCODING)


@lru_cache(maxsize=50_000)
def count_text(text: str) -> int:
    return len(_enc().encode(text))


def count_messages(messages: list[dict]) -> int:
    return sum(count_text(m["content"]) + PER_MESSAGE_OVERHEAD for m in messages)


def message_boundaries(messages: list[dict]) -> list[int]:
    """Cumulative token count at the end of each message."""
    out, total = [], 0
    for m in messages:
        total += count_text(m["content"]) + PER_MESSAGE_OVERHEAD
        out.append(total)
    return out
Enter fullscreen mode Exit fullscreen mode

Gotcha: tiktoken downloads its vocabulary on first use. In my sandbox, get_encoding("o200k_base") failed with a proxy 403 because the BPE file is fetched from a blob storage URL. CI runners and locked-down containers hit the same wall. The library reads TIKTOKEN_CACHE_DIR, names the cached file by the SHA-1 of the source URL, and checks the file’s SHA-256 against a hash pinned in the code. I copied o200k_base.tiktoken from the gpt-tokenizer npm package (v3.4.0), confirmed its SHA-256 matched the pinned 446a9538…, and put it in tiktoken_cache/. Bake that file into your image and the tokenizer never touches the network.

The ledger records one row per request with user, feature, model, usage and cost. Here is one request going through estimate → reserve → call → record:

== 1. estimate, reserve, call, record
estimated input tokens: 2031
worst-case cost to reserve: $0.004539
actual: Usage(input_tokens=2031, cached_tokens=0, cache_write_tokens=2031, output_tokens=220) -> $0.003639
Enter fullscreen mode Exit fullscreen mode

Note cache_write_tokens=2031: with implicit caching, the whole prompt was written to the cache at 1.25×. Keep that in mind for Step 4.

Step 2. Budgets per user and per feature (LLM cost management)

A ledger tells you who spent the money yesterday. A budget stops them today. The check runs before the call, against the worst case: every input token at the highest input rate and the model using all of max_tokens. After the call, the ledger records the actual cost. When the budget is tight, degrade to the cheap model before refusing.

def worst_case_cost(model: str, input_tokens: int, max_tokens: int) -> float:
    """Upper bound before the call: every input token billed at the cache-write
    rate (the highest input rate), and the model uses all of max_tokens."""
    usage = Usage(input_tokens=input_tokens, cache_write_tokens=input_tokens, output_tokens=max_tokens)
    return cost_usd(model, usage)
class Budget:
    """Daily USD limits. Check the worst case before the call, record the actual after."""

    def __init__(self, ledger: Ledger, per_user: float, per_feature: dict[str, float],
                 exempt=("system",)):
        self.ledger, self.per_user, self.per_feature = ledger, per_user, per_feature
        self.exempt = set(exempt)  # batch jobs and other non-user traffic

    def allows(self, user, feature, model, input_tokens, max_tokens) -> bool:
        worst = worst_case_cost(model, input_tokens, max_tokens)
        user_ok = user in self.exempt or self.ledger.by_user[user] + worst <= self.per_user
        feat_cap = self.per_feature.get(feature, float("inf"))
        feat_ok = self.ledger.by_feature[feature] + worst <= feat_cap
        return user_ok and feat_ok

    def pick_model(self, user, feature, preferred, fallback, input_tokens, max_tokens) -> str:
        """Degrade to the cheaper model before refusing the request."""
        for model in (preferred, fallback):
            if self.allows(user, feature, model, input_tokens, max_tokens):
                return model
        raise BudgetExceeded(f"{user}/{feature}: daily budget spent")
Enter fullscreen mode Exit fullscreen mode

Batch jobs run as user system and are exempt from per-user caps; give them a per-feature cap instead.

What a $1.00/day per-user budget did in the simulation:

per-user budget of $1.00/day:
  baseline           $63.78/day, degraded 110, rejected 3514, users at cap 12, u_heavy $1.00
  all optimizations  $10.61/day, degraded 200, rejected 192, users at cap 1, u_heavy $1.00
Enter fullscreen mode Exit fullscreen mode

On the unoptimized gateway the budget rejected 3,514 requests and capped 12 users, most of them your best customers. After the optimizations below, the same budget catches exactly one user: u_heavy, the log paster. A budget is a fuse, not a cost strategy. Set it after you have made normal traffic cheap, or it punishes normal users.

Step 3. Trim the chat history

Every chat turn resends the conversation. trim_history keeps the system prompt, the first user message, the latest message, and as many recent turns as fit in a token budget.

"""Context trimming: cap how much chat history goes into each request."""
from .tokens import count_messages


def trim_history(messages: list[dict], max_history_tokens: int, pin_first_user: bool = True):
    """Keep system messages, optionally the first user message (it usually states
    the problem), and as many of the latest turns as fit in max_history_tokens.
    The latest message is always kept, even if it alone exceeds the budget;
    the budget layer deals with oversized inputs."""
    system = [m for m in messages if m["role"] == "system"]
    rest = [m for m in messages if m["role"] != "system"]
    if not rest:
        return system
    pinned = [rest[0]] if pin_first_user and rest[0]["role"] == "user" and len(rest) > 1 else []
    tail_src = rest[len(pinned):]
    kept = [tail_src[-1]]
    budget = max_history_tokens - count_messages(pinned) - count_messages(kept)
    for m in reversed(tail_src[:-1]):
        cost = count_messages([m])
        if cost > budget:
            break
        kept.insert(0, m)
        budget -= cost
    return system + pinned + kept
Enter fullscreen mode Exit fullscreen mode

On this traffic trimming to 1,500 history tokens saved only 3.5%, because most chats are short and the 2,000-token policy prompt dominates every request. Long sessions (coding assistants, agents) are where trimming pays. The interesting result is the pin:

check: trim WITHOUT pinning the first user message -> accuracy 97.54%

Enter fullscreen mode Exit fullscreen mode

In the traffic, 20% of follow-ups refer back to the opening question. With the first user message pinned, accuracy stays at 99.96%. Without the pin, a plain sliding window drops it to 97.54%. Trimming also interacts with prompt caching: when old turns fall out of the window, the cached prefix for that conversation no longer matches.

Step 4. Prompt caching: the timestamp bug that costs more than no caching

Provider prompt caching is a discount on repeated prompt prefixes. The OpenAI guide says caching is on by default, the minimum cacheable prompt is 1,024 tokens for GPT-5.6 and later, reuse requires the entire rendered prefix to match, and you should put stable instructions first and timestamps and user-specific content at the end. In implicit mode the breakpoint sits at the end of the latest eligible message; explicit mode (prompt_cache_options.mode = "explicit" plus prompt_cache_breakpoint on content blocks) lets you choose where writes end, and with no breakpoints there are no writes. Anthropic’s version is explicit: you mark blocks with cache_control, the prefix covers tools, system and messages in that order, 5-minute writes cost 1.25× base input and 1-hour writes 2×, and reads cost 0.1× on most models (0.05× on Claude Opus 5.5 and Sonnet 5.5).

Now the common bug. A system prompt that starts with Current time: … changes on every request, so the prefix never matches. With write-priced caching that is worse than a missed discount: every request pays the 1.25× write rate for a cache entry nobody reads. Here are two different questions sent one minute apart, priced for the second request:

== 2. two different questions, one minute apart: what does the 2nd one cost?
timestamp first, implicit: input=2049 read=0 write=2049 input cost=$0.002561
timestamp last,  implicit: input=2047 read=0 write=2047 input cost=$0.002559
timestamp last,  explicit: input=2047 read=2020 write=0 input cost=$0.000128
caching off              : input=2049 read=0 write=0 input cost=$0.002049
Enter fullscreen mode Exit fullscreen mode

Input cost of the second request under four prompt layouts; only the explicit breakpoint after the policy gets the cached rate

Moving the timestamp to the end is not enough on its own. With an implicit breakpoint at the end of the prompt, the cache entry includes the user’s question, so a different question cannot reuse it. The fix is an explicit breakpoint right after the static policy (plus one at the end of the conversation, so the next turn of the same chat can reuse the history). In the gateway:

def _breakpoints(self, feature, msgs):
    if self.cfg.cache_mode == "implicit":
        return None                      # provider puts one at the end of the prompt
    if self.cfg.cache_mode == "explicit" and feature == "support_chat":
        return [0, len(msgs) - 1]        # after the policy, and after the latest turn
    return []                            # one-off prompts: no cache writes at all
Enter fullscreen mode Exit fullscreen mode

Over the whole day:

Prompt layout $/day
timestamp first, implicit caching (baseline) $80.37
same prompt, caching off $68.06
stable prefix, implicit caching (+ trimming) $44.05
stable prefix, explicit breakpoints (+ trimming) $34.68

The baseline pays 18% more than it would with caching off. The digests show the same effect in miniature: one-off 4,000-word documents written to the cache at 1.25× and never read. In explicit mode the gateway passes no breakpoints for them at all.

Step 5. Exact response cache

The cheapest request is the one you do not send. The exact cache keys on a normalized prompt, inside a namespace that includes the feature and the prompt version. Without the version, a prompt change keeps serving old answers until the cache expires.

def normalize(text: str) -> str:
    text = text.lower().strip()
    text = re.sub(r"[^\w\s]", "", text)
    return re.sub(r"\s+", " ", text)
class ExactCache:
    """Key = namespace + normalized text. The namespace must include the model
    and the prompt version, or a prompt change will keep serving old answers."""

    def __init__(self):
        self._d: dict[str, object] = {}

    def key(self, namespace: str, text: str) -> str:
        return hashlib.sha256(f"{namespace}\x1f{normalize(text)}".encode()).hexdigest()

    def get(self, namespace, text):
        return self._d.get(self.key(namespace, text))

    def put(self, namespace, text, value):
        self._d[self.key(namespace, text)] = value
Enter fullscreen mode Exit fullscreen mode

The cache only sees first-turn support questions and ticket classifications; follow-up turns depend on history and almost never repeat. It took the day from $34.68 to $31.52 and removed 4,226 calls. Most of those calls were repeated tickets that differed only in casing, but most of the dollars ($2.88 of $3.16) came from repeated support openers, because each one carries the 2,000-token policy and a 150 to 320-token answer.

Step 6. Semantic cache: find the threshold, then distrust it

A semantic cache returns a stored answer when a new question is similar enough to an old one. The threshold decides everything. To keep the demo offline, the embedding here is hashed character n-grams; a production system would use an embedding model, and the threshold problem is the same.

class SemanticCache:
    """Nearest cached question by cosine similarity; hit if similarity >= threshold.

    Embeddings here are hashed character n-grams so the demo runs offline. A real
    deployment would use an embedding model; the threshold problem is the same.
    """

    def __init__(self, threshold: float):
        self.threshold = threshold
        self._vec = HashingVectorizer(analyzer="char_wb", ngram_range=(3, 5),
                                      n_features=2**16, norm="l2", alternate_sign=False)
        self._rows: dict[str, sparse.csr_matrix] = {}  # namespace -> one row per cached question
        self._values: dict[str, list] = {}
        self.embed = lru_cache(maxsize=10_000)(self._embed)

    def _embed(self, text: str) -> sparse.csr_matrix:
        return self._vec.transform([normalize(text)]).astype(np.float32)  # 1 x n_features, L2-normalized

    def similarity(self, a: str, b: str) -> float:
        return float((self.embed(a) @ self.embed(b).T).toarray()[0, 0])

    def lookup(self, namespace, text):
        rows = self._rows.get(namespace)
        if rows is None:
            return None, 0.0
        sims = (rows @ self.embed(text).T).toarray().ravel()  # cosine, since rows are unit length
        best = int(np.argmax(sims))
        score = float(sims[best])
        if score >= self.threshold:
            return self._values[namespace][best], score
        return None, score

    def put(self, namespace, text, value):
        v = self.embed(text)
        old = self._rows.get(namespace)
        self._rows[namespace] = v if old is None else sparse.vstack([old, v], format="csr")
        self._values.setdefault(namespace, []).append(value)
Enter fullscreen mode Exit fullscreen mode

The sweep replays the 3,500 first-turn questions with the exact cache in front, so the semantic cache only sees exact misses:

closest pairs with DIFFERENT intents:
  0.841  'What happens to my data when I upgrade?' vs 'What happens to my data when I downgrade?'
  0.827  'shipping time to germany' vs 'shipping cost to germany'
  0.776  'How do refunds work for the monthly plan?' vs 'How do refunds work for the annual plan?'
  0.756  'Can I get a refund for my monthly plan?' vs 'Can I get a refund for my annual plan?'
same-intent pairs scoring below 0.40:
  0.390  'How do I delete my account?' vs 'Close my account permanently'
  0.390  'How much does shipping cost?' vs 'Why is shipping so expensive?'
  0.385  'export all my data to csv' vs 'Download a copy of all my data'
two different log pastes (15006 and 15006 tokens): 0.999

first-turn questions: 3500
threshold semantic hits false hits false share
     0.60          3271        149        4.6%
     0.70          3141         55        1.8%
     0.75          3049         52        1.7%
     0.80          2852         11        0.4%
     0.85          2558          5        0.2%
     0.90          2049          0        0.0%
     0.95          1026          0        0.0%
Enter fullscreen mode Exit fullscreen mode

Semantic cache threshold sweep: hits fall slowly while false hits drop to zero only at 0.90

Three things in that output are worth more than the hit counts:

  • The closest wrong pair scores higher than many right pairs. “upgrade” vs “downgrade” is 0.841. “How do I delete my account?” vs “Close my account permanently” is 0.390. No single threshold separates them.
  • False hits survive up to 0.85. Greetings and sign-offs (“hi, … thanks”) push near-miss pairs closer together. At 0.80 the cache gave 11 wrong answers; a user asking about an annual refund gets the monthly policy.
  • Long inputs all look alike. Two different 15,000-token log pastes scored 0.999. My first gateway version had no length gate and served one user’s log analysis to another. The fix is the SEMANTIC_MAX_TOKENS = 64 gate in the gateway: only short questions are eligible.

At 0.90 the semantic cache saved $0.91/day (−61.9% overall vs −60.8% before). That is small because first-turn answers are a small slice of the bill once the policy prompt is cached. Measure your own share before building one.

Step 7. LLM routing: cheap model first, escalate on low confidence

Routing is where the big money is. Send the request to the cheap model; if its confidence is below a threshold, send it again to the strong one. You pay for both calls on escalation.

"""Model routing: try the cheap model, escalate when it is not confident."""
CHEAP, STRONG = "gpt-6-luna", "gpt-6.1-terra"


def route(client, messages, *, max_tokens, now, task, threshold=0.7, cheap=CHEAP, strong=STRONG,
          breakpoints=None):
    """Returns (final_response, all_responses). Every response is billed,
    so an escalated request costs cheap + strong."""
    first = client.complete(cheap, messages, max_tokens=max_tokens, now=now, task=task,
                            breakpoints=breakpoints)
    if first.confidence >= threshold:
        return first, [first]
    second = client.complete(strong, messages, max_tokens=max_tokens, now=now, task=task,
                             breakpoints=breakpoints)
    return second, [first, second]
Enter fullscreen mode Exit fullscreen mode

The fake client produces a confidence score correlated with correctness. In a real system that signal comes from log probabilities, a verifier prompt, a classifier trained on your own graded answers, or a learned router. Research results point the same way: FrugalGPT reports matching the best individual LLM with up to 98% cost reduction using an LLM cascade, and RouteLLM reports cutting costs by over 2× in some cases while keeping response quality.

In the simulation:

routing: 3292 of 16743 routed requests escalated (19.7%)

Enter fullscreen mode Exit fullscreen mode

Routing cut the day from $30.61 to $15.42, the biggest single step after caching. It also cost 0.12 percentage points of accuracy (99.96% → 99.84%): some cheap answers were confidently wrong. Calibrate the threshold on graded samples from your own traffic, and track accuracy next to cost, or this step silently degrades the product.

Step 8. Batch API for work that can wait

The nightly digest does not need an answer in two seconds. The OpenAI Batch guide lists a 50% discount, a 24-hour completion window, and up to 50,000 requests or a 200 MB input file per batch; Anthropic’s Batch API is also 50% off both input and output tokens. In the gateway this is one flag, and the digest line dropped from $1.81 (after removing the useless cache writes) to $0.90.

The before/after table

prices: OpenAI API docs: Pricing (Standard and Batch tables), checked 2026-10-30
requests: 21995

scenario                            $/day vs base  calls   read  write accuracy
baseline: one model, full history   80.37    0.0%  21995   0.0%  98.8%   99.96%
+ trim history to 1,500 tokens      77.59   -3.5%  21995   0.0%  98.7%   99.96%
+ stable prompt prefix              44.05  -45.2%  21995  58.9%  39.8%   99.96%
+ explicit cache breakpoints        34.68  -56.8%  21995  74.7%  21.6%   99.96%
+ exact cache                       31.52  -60.8%  17769  72.1%  24.2%   99.96%
+ semantic cache (0.90)             30.61  -61.9%  17043  71.1%  25.1%   99.96%
+ routing (escalate < 0.7)          15.42  -80.8%  20335  67.8%  29.3%   99.84%
+ batch API for digests             14.52  -81.9%  20335  67.8%  29.3%   99.84%
Enter fullscreen mode Exit fullscreen mode

LLM cost per day after each optimization step, from $80.37 to $14.52

By feature, baseline vs final:

baseline vs final, by feature ($/day):
  classify_ticket     0.96 ->   0.08
  support_chat       77.32 ->  13.54
  nightly_digest      2.09 ->   0.90
Enter fullscreen mode Exit fullscreen mode

At 30 days that is roughly $2,411 vs $436 a month for the same traffic. Order matters when you read the table: each row includes every row above it, so routing’s share would look different if it went first.

Gotchas

  • Caching is on by default and can cost you. Look at write tokens in usage, not only cached tokens. A dynamic value at the top of the prompt turns every request into a paid write.
  • Implicit breakpoints do not share a system prompt across different questions. Put an explicit breakpoint after the static part.
  • Cache keys need a prompt version. Otherwise a fixed prompt keeps serving old answers.
  • Semantic caches need a length gate and a per-domain threshold. Run a sweep with labeled pairs that differ by one word.
  • Budgets check the worst case, not the average. Reserve against max_tokens and the highest input rate, then record the actual cost.
  • Pin the first user message when trimming. A sliding window cost 2.4 points of accuracy here.
  • Your token estimate is not the bill. Reconcile the ledger with usage and with the provider invoice.

Production checklist

  • One gateway for every LLM call, with user, feature and prompt version on each request.
  • A ledger table in your database; a daily report by feature and by user.
  • Price table in config with a “checked on” date, reviewed when providers announce changes.
  • Prompt layout reviewed for cache friendliness: static first, dynamic last, explicit breakpoints.
  • Accuracy tracked next to cost for every routing or caching change.
  • Budgets with degrade-then-refuse behavior and a clear error for the user.
  • Batch API for anything that can wait hours.

Wrapping up

The biggest wins here came from boring places: a timestamp in the wrong spot, a missing cache breakpoint, and sending easy requests to an expensive model. The semantic cache, which gets the most attention, was the smallest line in the table. Measure per request first, then optimize the line items that are actually large.

What does your LLM cost breakdown look like: is it dominated by prompts, outputs, or one heavy user?

Top comments (0)