Postmortem: The Stale-Response Incident That Broke Our Review Bot

A broken PR slipped through. The model was fine. Our cache was not. This postmortem covers timeline, root causes, and the durable fix we now apply to every AI-backed job.

We run our review bot on MonkeyCode's free server using its free model access. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Timeline

All times UTC.

• 01:10 Deployed v1.8 of the review bot to staging.
• 01:15 Added a Redis cache for all model responses. TTL was one hour.
• 03:22 Production received a security-sensitive PR.
• 03:23 Bot returned a cached approval. The cached answer was two hours old.
• 03:25 Auto-merge rule approved the PR.
• 04:47 Integration tests failed. The change broke a payment edge case.
• 06:15 Rolled back the PR and disabled auto-merge.
• 09:30 Root cause confirmed: stale cache poisoning.

Contributing Factors

Cache Key Missed the Environment

The cache key was just the prompt string. Staging and production share the same model prompts. A staging request with a truncated context populated the cache. Production later hit that same key.

The fix was obvious in hindsight. Cache keys must include the environment, model version, and a context hash.

Retry Loop Amplified the Damage

When the model provider rate-limited us, the bot retried every five seconds. On failure, it fell back to whatever was in the cache. That makes sense in a simple design. In practice it converted transient rate limits into silent stale outputs.

The retry loop had no circuit breaker state. The free server we used has an ephemeral process lifecycle. Any restart wiped the in-memory breaker. The bot charged ahead after every restart.

No Cache-To-Cache Parity Tests

We tested the model, not the cache. All green tests used fresh responses. The stale path only appeared under a specific key collision. We never wrote a test for that.

The Durable Fix

We replaced the naive cache with three layers.

1. Context-Aware Cache Keys

import hashlib

def cache_key(env: str, model: str, context: str) -> str:
    payload = f"{env}:{model}:{hashlib.sha256(context.encode()).hexdigest()}"
    return f"ai-response:{payload}"

Now staging cannot poison production. The prompt alone no longer decides the key.

2. Bounded TTL With Refresh Jitter

import random

def ttl_seconds(base_ttl: int = 3600) -> int:
    jitter = random.randint(-300, 300)
    return max(60, base_ttl + jitter)

Jitter prevents thundering herd at TTL expiry. We also cap TTL at 30 minutes for review bots. Old code reviews are worse than none.

3. Circuit Breaker With Persistent State

class AIBreaker:
    def __init__(self, redis):
        self.redis = redis
        self.key = "ai:breaker:state"

    def is_open(self) -> bool:
        return self.redis.get(self.key) == b"open"

    def open(self, ttl=120) -> None:
        self.redis.set(self.key, b"open", ex=ttl)

The breaker state lives in Redis. A server restart no longer resets it. We stop calling the model and stop reading the cache when the breaker is open.

Testing the New Behavior

We added a regression test that simulates the entire incident. You can reproduce it with this plan:

1. Start a local Redis instance.
2. Populate a cache entry with a fake model response.
3. Change the environment key from staging to production.
4. Confirm the production call misses the cache.
5. Set the breaker to open.
6. Restart the worker process.
7. Confirm the breaker is still open.

Here is the core test in Python:

def test_staging_does_not_poison_production():
    staging_key = cache_key("staging", "gpt-x", "patch context")
    prod_key = cache_key("production", "gpt-x", "patch context")
    assert staging_key != prod_key

def test_breaker_survives_restart():
    breaker = AIBreaker(redis)
    breaker.open(ttl=120)
    new_breaker = AIBreaker(redis)
    assert new_breaker.is_open()

The test suite now catches any regression in key isolation or persistent state.

What We Changed in Operations

We added a cache miss metric and a stale-hit alert. Every cached response now carries a generated_at timestamp. Alerts fire when the timestamp is older than 15 minutes.

We also removed the fallback-to-cache behavior on retry. If the model call fails and the breaker is closed, the job fails loudly. A failed job is safer than a confident stale answer.

Who Should Not Use This Approach

Do not copy this fix if you have no observability around cache hits. A cache with no metrics is a black box. You need at least a hit-rate counter and a stale-response log.

Do not use cache fallback for security-critical decisions. Approval gates should never silently reuse old model outputs. If you cannot tolerate a wrong answer, skip the cache entirely.

Lessons

Caching AI responses is not free. It trades compute cost for consistency risk. You must treat cache keys as security boundaries.

Free model tiers and free servers are useful. They force you to think about quotas and ephemeral state. But they do not excuse sloppy cache design.

Our review bot is now boring. It misses cache keys correctly. It opens breakers persistently. It fails loudly when it should.

That boringness is the best outcome of a painful postmortem.

If you run AI-assisted code review, think first about your cache key. The model is the easy part. The cache is where trust goes to die.