DEV Community: Hanlin Xiang

The #3 Production Killer in Your LiteLLM Setup: Key Cache Invalidation (and How to Fix It)

Hanlin Xiang — Fri, 19 Jun 2026 14:18:28 +0000

This is the pitfall that cost me 3 hours at 2 AM. If you're running LiteLLM Proxy in production, it will hit you too — usually at the worst possible time.

What Happened

I run LiteLLM Proxy + New API in front of 18 provider channels. One night, I rotated an API key for a provider that had been flagged for unusual spending.

Standard procedure:

Generate new key in provider dashboard
Update config.yaml with new key
Run litellm --config config.yaml --reload

The reload succeeded. No errors. The config showed the new key. I went to sleep.

The next morning, the old key was still being used. Every single request was still authenticating with the rotated-out key. The provider's dashboard showed traffic from both keys — the new one (from config validation) and the old one (from actual API calls).

Why It Happens

LiteLLM caches API keys in-memory for performance. When you --reload, the config is reloaded, but the key store is not purged. The worker process holds the old keys in a dictionary that persists across config reloads.

This means:

config.yaml shows the new key ✅
litellm --model_cost_map shows the new key ✅
The actual HTTP requests use the old key ❌

You won't notice until the old key expires or is revoked — at which point every request to that provider starts returning 401, and your fallback chain kicks in, routing traffic to your most expensive model.

The Fix

Option 1: Purge the cache manually (no downtime)

curl -X POST http://localhost:4000/cache/purge \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY"

This clears the in-memory key cache. The next request will pull the key from the freshly reloaded config.

Option 2: Use Redis for shared key state (recommended for multi-worker)

Set REDIS_HOST in your environment:

# docker-compose.yml
environment:
  - REDIS_HOST=redis://redis:6379
  - REDIS_CONNECTION_POOL_SIZE=5

With Redis, keys are stored externally. A config reload triggers a Redis key update, and all workers pick it up immediately. No stale keys.

Option 3: Restart the worker (downtime: 2-5 seconds)

docker restart litellm-proxy

Brute force, but guaranteed to work. Use this if you're in a hurry and can afford a brief blip.

How to Detect It Before Users Do

Add this to your monitoring — a simple script that checks whether the key in config matches the key actually being used:

# Check which key is being used for a specific model
curl -s http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_API_KEY" \
  -d '{"model": "openai/gpt-4o", "messages": [{"role": "user", "content": "test"}], "max_tokens": 1}' \
  | jq '.usage'

# Compare with the key in config
grep "api_key:" config.yaml | head -1

If the provider's response includes a x-api-key-id header (OpenAI does), you can verify which key was used without guessing.

The Bigger Picture

Key cache invalidation is Pitfall #3 in my production survival map. There are 4 more deployment pitfalls and 3 hidden cost traps that I documented after 6 months of running this stack:

503 on every request after adding a provider — model name mismatch
Costs 3× higher than expected — fallback chain hits expensive models by default
Keys rotated but old ones still work ← this one
Streaming responses cut off mid-token — Nginx/Cloudflare buffering
New API channels show "insufficient quota" with balance > 0 — weight = 0 by default

Each of these took me 1-2 hours to diagnose in production. The full one-page reference card with all 5 pitfalls, 3 cost traps, a failure decision tree, and a pre-launch security checklist is available here:

👉 AI API Gateway Pitfall Map — $9

It's the page you print and pin next to your monitor — because when your gateway goes down at 2 AM, you won't be reading a 40-page guide.

5 Pitfalls I Hit Running LiteLLM Proxy in Production (with a 1-page survival map)

Hanlin Xiang — Wed, 17 Jun 2026 12:09:44 +0000

I've spent the last 6 months running an 18-channel LLM gateway in production — LiteLLM Proxy backed by Redis and PostgreSQL, routing traffic across OpenAI, Anthropic, Google, DeepSeek, and several smaller providers. What started as a weekend project turned into a 24/7 operation serving multiple AI agents and internal tools.

This post covers the 5 pitfalls that hit me hardest, with real error examples and the fixes that worked. If you're running LiteLLM Proxy (or considering it), these are the things I wish someone had told me before I went to production.

Pitfall #1: Silent OOM (Memory Leak + No systemd MemoryMax)

LiteLLM has a known memory leak under high concurrency. Without a hard memory limit, the process will eat all available RAM until the kernel's OOM-killer takes it down — usually at 3 AM.

# The symptom: requests start timing out intermittently
# Check dmesg for the kill signal
import subprocess
result = subprocess.run(["dmesg", "-T"], capture_output=True, text=True)
for line in result.stdout.split("\n"):
    if "oom" in line.lower() and "litellm" in line.lower():
        print(f"FOUND: {line}")

# The fix: systemd unit with MemoryMax
# /etc/systemd/system/litellm.service
# [Service]
# ExecStart=/usr/bin/litellm --config /opt/litellm/config.yaml --port 4000
# MemoryMax=4G
# Restart=always
# RestartSec=10

The fix is simple but easy to miss: set MemoryMax=4G (or whatever your server can spare) in your systemd unit. The proxy will restart cleanly instead of being force-killed.

Pitfall #2: Key Cache Miss (OpenAI 8min Cache vs 24h Cache Key)

This was the single most painful bug I encountered. I rotated a provider API key through the config and ran litellm --reload. The config file updated, but LiteLLM caches keys in-memory. The old key kept getting used for hours.

# What happened: silent 401s that looked like "provider outage"
# The config was correct, but the in-memory cache wasn't purged

# WRONG: this only reloads config, not the key store
# litellm --reload

# RIGHT: purge the cache explicitly
import requests
requests.post(
    "http://localhost:4000/cache/purge",
    headers={"Authorization": "Bearer YOUR_MASTER_KEY"}
)

# Or just restart the worker entirely
# If using REDIS_HOST for shared state, flush that too:
# redis-cli -h $REDIS_HOST FLUSHDB

The key insight: --reload refreshes the config file but does NOT purge the in-memory key cache. You need to either hit /cache/purge or restart the worker. If you're using Redis for shared key state, flush that too.

Pitfall #3: Retry Storm (4xx Retries Cause Rate-Limit Avalanche)

LiteLLM retries num_retries=3 by default. A single failed call becomes 3x the token spend. Worse: on 4xx errors (which should NOT be retried), the retry logic can trigger rate-limit cascades.

# The problem: default config retries everything
# config.yaml (BAD)
# litellm_settings:
#   num_retries: 3  # This retries even 4xx errors!

# The fix: retry only 5xx, use fallbacks for 4xx
# config.yaml (GOOD)
litellm_settings:
  num_retries: 1
  retry_policy:
    InternalServerError:
      retries: 1
    RateLimitError:
      retries: 0  # Don't retry rate limits — use fallbacks instead

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      fallbacks: ["anthropic/claude-3.5-sonnet"]

Set num_retries: 1 for non-critical paths. Use fallbacks (cheapest-first) instead of retries for cost control. A retry storm on a rate-limited provider can 3x your spend in 5 minutes.

Pitfall #4: Cost Unobserved (Multi-Provider Routing Weights)

When you route across multiple providers, LiteLLM's default fallbacks are sequential — not cost-sorted. One upstream failure can route all traffic to your most expensive model.

# The problem: fallbacks hit the most expensive model first
# config.yaml (BAD)
model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      fallbacks: ["openai/gpt-4o-mini", "anthropic/claude-3.5-sonnet"]
      # If gpt-4o fails, it tries mini (cheap) then sonnet (expensive)
      # But if mini also fails, ALL traffic goes to sonnet

# The fix: sort fallbacks cheapest-first + set max_budget per team
# config.yaml (GOOD)
litellm_settings:
  max_budget: 100.0  # Daily budget cap in USD
  budget_duration: "1d"

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      allowed_fails: 3
      fallbacks: ["openai/gpt-4o-mini"]  # Only fall back to cheaper models

Monitor per-provider spend daily. The LiteLLM UI shows cost breakdowns, but only if you've configured master_key and database_url properly.

Pitfall #5: Metric Blindness (Incomplete Prometheus Metrics)

LiteLLM's built-in Prometheus metrics don't cover per-provider latency percentiles or cost attribution. You're flying blind on the most important signals for production operations.

# What LiteLLM exposes by default:
# - litellm_requests_total
# - litellm_request_duration_seconds (aggregate, not per-provider)
# - litellm_spend_total (only if database is configured)

# What's MISSING:
# - Per-provider P95/P99 latency
# - Per-provider error rate
# - Per-team cost breakdown
# - Cache hit/miss ratio

# The fix: add a custom middleware to emit per-provider metrics
from litellm.integrations.custom_logger import CustomLogger
import prometheus_client as prom

provider_latency = prom.Histogram(
    'litellm_provider_latency_seconds',
    'Latency by provider',
    ['provider', 'model']
)

class PerProviderMetrics(CustomLogger):
    def log_success_event(self, kwargs, response_obj, start_time, end_time):
        provider = kwargs.get("litellm_params", {}).get("custom_llm_provider", "unknown")
        model = kwargs.get("litellm_params", {}).get("model", "unknown")
        latency = (end_time - start_time).total_seconds()
        provider_latency.labels(provider=provider, model=model).observe(latency)

Without per-provider metrics, you can't tell if DeepSeek is slow today or if OpenAI is throttling you. Add a custom logger to fill the gap.

The Solution: A 1-Page Survival Map

After hitting all 5 of these pitfalls in production (and losing too many weekends to debugging), I compiled everything into a single-page survival map. It covers:

All 5 deployment pitfalls with symptoms, root causes, and fixes
3 hidden cost traps (retry amplification, embedding tax, idle-connection keep-alive)
A failure decision tree for any error code you'll see
A pre-launch security checklist
Copy-paste diagnostic commands

The full map is available here: https://payhip.com/b/S96bB

It's $9, no email signup, no affiliate. Just the thing I wish I had when I started.

Have you hit any of these pitfalls? Or did I miss something that's bitten you? Drop a comment — I'll be responding to everyone. You can find me at @ai-gateway-veteran on Reddit and X.