Your Semantic Cache Has a Health Problem. Nothing Checks It.

Here's the situation: every LLM gateway now ships a semantic cache. GPTCache, LangChain's caches, Redis vector lookup — they all do the same job: store a response, look it up later by embedding similarity, skip a costly LLM call.

But they only operate the cache. Store. Lookup. Evict. Nobody diagnoses it.

That leaves three silent failure modes running unobserved in production:

1. Cache pollution. A lightly rewritten prompt — "Capital of France?" vs "What is the capital of France?" — matches a cached response it was never meant to match. The cache serves the wrong answer, fast, confidently, and nobody notices because there's no error, just a subtly wrong reply.
2. Expiry waste. Entries expire before the traffic that would hit them arrives. Every miss is a full-price LLM call that a longer TTL would have avoided.
3. Key fragmentation. The same prompt, after sloppy normalization, gets stored under three different keys. Triple the storage, a third of the hit rate, and it looks like a capacity problem instead of a bug.

I built a tool that performs a checkup on the cache instead of just using it.

What I built

cache-doctor — a zero-dependency Python CLI (stdlib only, Python ≥ 3.9) that reads a semantic-cache event log and produces a health report plus severity-ranked optimization suggestions.

pip install cache-doctor
cache-doctor analyze cache-log.jsonl

Feed it JSONL — one event per line:

{"ts": "2026-10-11T01:00:01Z", "prompt": "What is the capital of France?",
 "cache_key": "k001", "hit": false, "miss_reason": "no_key",
 "threshold": 0.9, "ttl_seconds": 3600}
{"ts": "2026-10-11T01:00:22Z", "prompt": "Capital of France?",
 "cache_key": "k001", "hit": true, "similarity": 0.91,
 "threshold": 0.9, "ttl_seconds": 3600, "cache_age_seconds": 60}

And you get this:

cache-doctor  health report
============================================
events analyzed :         16
cache hits      :          6     37.5%  [###########-------------------]
cache misses    :         10
similarity thr. :        0.9

miss reasons:
  expired             2    20.0%
  low_similarity      3    30.0%
  no_key              5    50.0%

pollution signals:
  borderline hits :      3  ( 50.0% of hits)
  wrong hits      :      1   <- flag_wrong=true
  near misses     :      3  ( 30.0% of misses)

policy signals:
  expired misses  :      2  ( 20.0% of misses)
  stale hits      :      1  ( 16.7% of hits)
  frag. groups    :      1  (3 requests)

optimization suggestions
============================================
1. [!!] CRITICAL   1 cache hit(s) served the WRONG response
   -> Raise the similarity threshold above 0.9, require stricter
      embedding agreement (e.g. rerank top-3 candidates)...

2. [ *] MEDIUM    1 prompt group(s) stored under multiple keys
   -> Normalize prompts before keying: trim, collapse whitespace,
      casefold, and strip volatile prefixes...

How it detects pollution

Two layers:

• Borderline hits. Any hit whose similarity lands within +0.05 of your threshold is flagged. That's the exact band where "almost the same prompt" slips through — legitimate paraphrases you want, and false positives you don't. A high borderline share means your threshold is a coin flip.
• Ground truth. If your eval pipeline ever confirms a hit served the wrong response, log "flag_wrong": true. One of those is a CRITICAL finding, not a metric — because a polluted cache doesn't degrade gracefully, it lies.

It also tracks near misses (similarity just below the threshold — your threshold may be too strict), expired-miss share (your TTL is too short), stale serving (hits from entries past 90% of TTL), and rolling hourly hit rates so you can see when the cache degraded, not just that it did.

Why not just use the cache product's own dashboard?

Because cache products report operations: hits, misses, evictions. They don't tell you that 50% of your hits are borderline, that three keys hold the same prompt, or that 20% of your misses were expired entries that would have hit with a longer TTL. cache-doctor is to a semantic cache what a database health monitor is to a database — the cache is the patient, this is the checkup instrument.

Try it

pip install cache-doctor
cache-doctor analyze examples/sample-log.jsonl   # from the repo

cache-doctor analyze --json for machine-readable metrics, cache-doctor suggest --log for suggestions only, --state to snapshot metrics between runs.

Repo: https://github.com/hahahahahahahahah6/cache-doctor · PyPI: cache-doctor · MIT licensed.