Our cache hit rate was 90 percent and the bill still climbed

The dashboard said we were serving nine of every ten requests from cache. The invoice said we were paying more every week. Both numbers were correct. The hit rate was counting requests and the bill was counting dollars, and those are not the same thing once your traffic is lopsided.

The response cache went in for exactly the reason you would expect. Repeated prompts were hitting the model over and over, so we keyed on the prompt, stored the completion, and served the stored answer on a match. Within a week the hit-rate panel settled around 90 percent and stayed there. Nice flat line. Everyone moved on.

Then the monthly spend kept creeping. Not a spike, a slope. Up a bit, up a bit more, up again. The cache was clearly working (90 percent of requests were free), so nobody looked at it as the culprit. That was the mistake. The cache was doing exactly what the panel measured. We had picked the wrong thing to measure.

What the hit rate was actually counting

A hit rate is a ratio of requests. Ninety percent hit means nine of ten requests were served from the store and one went to the model. What that number never told us is which requests were in which bucket.

Our traffic was not uniform. The bulk of it was short, cheap, repetitive calls: the same handful of classification and lookup prompts, hammered thousands of times an hour. Those cached beautifully. They were also nearly free per call. Meanwhile a thin slice of traffic was long-context work, tens of thousands of tokens of document stuffed into the prompt, and those were mostly unique. They missed the cache almost every time.

So the 90 percent hit rate was 90 percent of the cheap requests and almost none of the expensive ones. The cache was serving the traffic that barely cost anything and passing through the traffic that cost real money. Counted by request, the cache looked like it was carrying us. Counted by dollars spent, it was barely touching the bill. The panel only knew how to count requests, so it never showed us the second number.

What it actually cost

Put rough numbers on it. Say a cheap call is 200 tokens and a long-context call is 30,000 tokens, so the expensive one costs roughly 150 times more. If 95 percent of requests are cheap and 5 percent are long-context, and the cheap ones hit cache 94 percent of the time while the long ones hit 8 percent of the time, the request-weighted hit rate still lands around 90 percent. The dollar-weighted hit rate, dollars served from cache over total dollars, was under 15 percent.

That is the whole gap. Ninety percent of requests free, and we were still on the hook for something like 85 percent of the money. The panel we trusted only counted requests, and requests were not what showed up on the invoice.

Two more things the cache was quietly doing wrong

Once we started looking, two smaller problems fell out of the same investigation.

The keys were too strict. We keyed on the raw prompt string. A trailing space, a reordered pair of retrieved chunks, a timestamp injected into the system prompt, any of these produced a different key and a forced miss on what was semantically the same request. We were busting the cache on prompts that should have collided. Normalizing the key (strip whitespace, sort the parts that are order-insensitive, drop the volatile fields that do not change the answer) recovered a real chunk of hits on the traffic that mattered.

The cheap calls were inflating the number we bragged about. Because the tiny repetitive calls dominated the request count and cached almost perfectly, they dragged the headline hit rate up toward 90 no matter what the expensive traffic did. The one metric on the wall was structurally incapable of showing us the miss cost, because the misses were rare by count and huge by dollar.

The metric we should have had from day one

The fix is to weight the hit rate by cost instead of by request. Every lookup carries a dollar estimate. Sum the dollars you served from cache, sum the total dollars you would have spent, and divide. That number moves when an expensive request misses, which is exactly when you want it to move.

from dataclasses import dataclass, field

# price per 1k tokens for the model you are caching in front of.
# use your real numbers; these are placeholders.
INPUT_PER_1K = 0.0005
OUTPUT_PER_1K = 0.0015

def dollar_cost(prompt_tokens: int, output_tokens: int) -> float:
    return (prompt_tokens / 1000) * INPUT_PER_1K + (output_tokens / 1000) * OUTPUT_PER_1K

@dataclass
class CostWeightedCacheStats:
    dollars_from_cache: float = 0.0   # cost we avoided by serving a hit
    dollars_total: float = 0.0        # cost we would have paid with no cache
    hits: int = 0
    misses: int = 0
    miss_costs: list = field(default_factory=list)  # per-miss dollar cost

    def record_hit(self, prompt_tokens: int, output_tokens: int) -> None:
        c = dollar_cost(prompt_tokens, output_tokens)
        self.dollars_from_cache += c
        self.dollars_total += c
        self.hits += 1

    def record_miss(self, prompt_tokens: int, output_tokens: int) -> None:
        c = dollar_cost(prompt_tokens, output_tokens)
        self.dollars_total += c
        self.miss_costs.append(c)
        self.misses += 1

    @property
    def request_hit_rate(self) -> float:
        total = self.hits + self.misses
        return self.hits / total if total else 0.0

    @property
    def dollar_hit_rate(self) -> float:
        # the number that actually tracks the bill
        return self.dollars_from_cache / self.dollars_total if self.dollars_total else 0.0

    def miss_cost_percentiles(self):
        # where the missed money lives: is it the long tail?
        if not self.miss_costs:
            return {}
        s = sorted(self.miss_costs)
        def pct(p):
            return s[min(len(s) - 1, int(p / 100 * len(s)))]
        return {"p50": pct(50), "p90": pct(90), "p99": pct(99), "max": s[-1]}

Wire it around the cache lookup and both numbers fall out of the same code path.

stats = CostWeightedCacheStats()

def cached_call(req, cache):
    key = normalize_key(req.prompt)     # strip, sort order-insensitive parts, drop volatile fields
    hit = cache.get(key)
    if hit is not None:
        stats.record_hit(req.prompt_tokens, hit.output_tokens)
        return hit
    resp = call_model(req)
    stats.record_miss(req.prompt_tokens, resp.output_tokens)
    cache.set(key, resp)
    return resp

# emit both, side by side, so nobody trusts the wrong one again
def emit_metrics():
    return {
        "cache.request_hit_rate": round(stats.request_hit_rate, 3),
        "cache.dollar_hit_rate": round(stats.dollar_hit_rate, 3),
        "cache.miss_cost": stats.miss_cost_percentiles(),
    }

When we put those two lines on the same panel, the gap was undeniable. Request hit rate near 90, dollar hit rate scraping the bottom. The miss-cost percentiles made it concrete: the p99 miss was hundreds of times the median miss. All the money was in the tail, and the tail was invisible on the old graph.

Normalizing the keys pulled the dollar hit rate up, because some of that expensive traffic was more repetitive than the strict key let us see. But the honest conclusion was that a response cache in front of mostly-unique long-context calls was never going to save much. Knowing that stopped us from over-investing in a cache that could not move the bill, and pointed us at prompt trimming and shorter retrieved context instead, which did.

What I'd page on

Different checklist from the autoscaling write-up. This one is about a cache lying to you by measuring the wrong axis.

• Dollar-weighted hit rate. Dollars served from cache over total dollars. This is the headline. Warn if it drops below your target while request hit rate stays high, because that divergence is the exact failure in this post. Request hit rate alone is a vanity metric here.
• Request hit rate vs dollar hit rate, on one panel. Never show one without the other. The wider the gap between them, the more spend the request number is hiding.
• Miss cost distribution, p50 / p90 / p99. Where the unserved money lives. A p99 miss orders of magnitude above the median means your expensive traffic is bypassing the cache, and that is where to spend engineering time.
• Key-collision rate. Fraction of lookups where a normalized key would have hit but the raw key missed. A rising value means your keys are too strict and you are busting cache on semantically identical prompts. Alert when it climbs.
• Cost per served request, hourly. Total spend over requests served. Flat is healthy. Rising while the request hit rate holds steady is the surprise invoice forming, and it is invisible on any request-count panel.
• Cache dollar savings, absolute per day. Not a rate, a number: dollars the cache actually kept off the bill today. If it stops growing while traffic grows, the cache has stopped earning its keep and it is time to look upstream at prompt and context size.