Workshop: Prove a Response Cache Is Safe in 75 Minutes Before You Ship It

Most response caches in AI features fail for reasons that have nothing to do with Redis. They fail because the cache key ignores one input that actually changes the answer, and nobody wrote a test that could catch it. This workshop gives a team of two to six engineers a 75-minute path from "we should cache this" to a signed-off safety argument, using a runnable Python harness and a short adversarial test list.

The framing matters more than the tooling. As teams wire agents and scripted workflows into CI, the same prompt gets replayed dozens of times per pipeline run, so caching stops being an optimization and becomes a correctness control. A wrong hit is worse than a slow miss, because it silently returns a confident answer to a question nobody asked.

What the workshop produces

By minute 75 the room should hold four artifacts, not opinions:

1. A written cache key specification that lists every input able to change the output.
2. A fingerprint() function plus a CachedClient wrapper with injectable clock and upstream.
3. A seven-case adversarial test file that fails when the key is incomplete.
4. A filled-in measurement worksheet with hit rate, upstream call count, and p50 latency from your own run.

Pre-work (10 minutes, before the session)

Each participant picks one real request class from their own codebase, for example "ticket triage classification" or "changelog summary". They bring the actual system prompt, one sample user message, and the parameter values. No API key is required to start, because the harness below runs against a fake upstream.

Timing plan for 75 minutes

Block	Minutes	Activity	Output
1	0-8	Frame the failure mode: enumerate one wrong-hit incident from memory	Shared hazard list
2	8-22	Key design exercise: participants list every output-changing input, then compare lists	Cache key spec per team
3	22-42	Build fingerprint() and CachedClient, wire the fake upstream	Running code
4	42-60	Adversarial tests: each pair writes one test that must fail on a naive key	Seven-case test file
5	60-70	Measure with the replay harness, fill the worksheet	Filled numbers
6	70-75	Debrief: cache / no-cache decision per request class	Decision table

Keep block 2 strictly timed. The most common outcome is that two pairs disagree about whether retrieval context belongs in the key, and that disagreement is the lesson.

The worked example students can rerun

The code below uses only the standard library. Save it as lab/cache_lab.py and run python -m pytest lab -q; the first run should pass, and the deliberately broken variant in Exercise 3 should not.

# lab/cache_lab.py
import hashlib
import json
import time
from dataclasses import dataclass

TEMPLATE_VERSION = "2026-11-09.1"
CACHEABLE_FIELDS = ("system", "user", "context", "temperature", "max_tokens", "locale")


@dataclass(frozen=True)
class Request:
    system: str
    user: str
    context: str = ""
    temperature: float = 0.0
    max_tokens: int = 512
    locale: str = "en"


def canonical(obj) -> str:
    return json.dumps(obj, sort_keys=True, ensure_ascii=False, separators=(",", ":"))


def fingerprint(req: Request) -> str:
    payload = {name: getattr(req, name) for name in CACHEABLE_FIELDS}
    payload["template_version"] = TEMPLATE_VERSION
    return hashlib.sha256(canonical(payload).encode("utf-8")).hexdigest()


def is_cacheable(req: Request) -> bool:
    return req.temperature == 0.0


class CachedClient:
    def __init__(self, upstream, ttl_s=900.0, clock=time.monotonic):
        self.upstream = upstream
        self.ttl_s = ttl_s
        self.clock = clock
        self._store = {}
        self.stats = {"hit": 0, "miss": 0, "bypass": 0, "upstream_calls": 0}

    def complete(self, req: Request) -> str:
        if not is_cacheable(req):
            self.stats["bypass"] += 1
            self.stats["upstream_calls"] += 1
            return self.upstream.complete(req)
        key = fingerprint(req)
        now = self.clock()
        entry = self._store.get(key)
        if entry is not None and entry[0] > now:
            self.stats["hit"] += 1
            return entry[1]
        self.stats["miss"] += 1
        self.stats["upstream_calls"] += 1
        result = self.upstream.complete(req)  # raise => nothing is stored
        self._store[key] = (now + self.ttl_s, result)
        return result

The seven adversarial tests

Write these as pairs; each pair owns one test and must show it failing against a naive key of sha256(user_prompt).

1. Idempotence. Same request twice yields one upstream call.
2. Sampling bypass. temperature=0.5 must never be served from cache.
3. Template drift. Bumping TEMPLATE_VERSION must produce a miss.
4. Key ordering. Reformatted JSON with identical semantics must produce a hit.
5. Context sensitivity. Changing one retrieved chunk must produce a miss.
6. TTL expiry. Advancing the injected clock past ttl_s must produce a miss.
7. Error isolation. An upstream exception must propagate and leave the store unchanged.

# lab/test_cache_lab.py
import pytest
from cache_lab import CachedClient, Request


class FakeUpstream:
    def __init__(self):
        self.calls = 0

    def complete(self, req):
        self.calls += 1
        return f"answer-{self.calls}"


class FakeClock:
    def __init__(self):
        self.now = 0.0

    def __call__(self):
        return self.now


def test_context_sensitivity():
    up, clock = FakeUpstream(), FakeClock()
    client = CachedClient(up, ttl_s=60, clock=clock)
    base = Request(system="s", user="u", context="chunk-A")
    client.complete(base)
    client.complete(Request(system="s", user="u", context="chunk-B"))
    assert up.calls == 2


def test_sampling_bypass():
    up = FakeUpstream()
    client = CachedClient(up, clock=FakeClock())
    req = Request(system="s", user="u", temperature=0.7)
    client.complete(req)
    client.complete(req)
    assert up.calls == 2
    assert client.stats["bypass"] == 2

Exercise: measure, then argue

Appendix B of the lab is a blank worksheet. Fill it from your own run rather than copying numbers from any article, including this one:

• Replay N=200 logged requests from your own traffic sample through CachedClient.
• Record hit, miss, bypass, and upstream_calls from client.stats.
• Record wall-clock p50 and p95 for hit versus miss paths on the same machine.
• Note the TTL you chose and the business reason for it.

Two honest observations usually emerge. Deterministic extraction tasks cache extremely well, while anything with a free-form chat shape shows a low hit rate that does not justify the operational risk.

Decision table to close the session

Request class	Cache?	Key must include	Main risk
Classification at temperature=0	Yes	Template version, locale, full input	Silent label drift after prompt edits
Summary over retrieved chunks	Yes, short TTL	Every chunk text or its stable hash	Stale context after re-index
Free-form chat with user polarity	No	Not applicable	Wrong-tone replies attributed to the user
Any request containing PII	Only with scoped keys and a deletion policy	Tenant id and data version	Cross-tenant leakage through shared storage

If a participant cannot name what breaks when the key is wrong, they should not enable the cache. That is a legitimate outcome of the workshop.

Where a free endpoint fits, and what it does not solve

Block 4 needs live calls only if you want real latency numbers; the fake upstream is enough for correctness work. When you do want live numbers, paying per call for a teaching lab is wasteful, and this is the narrow place where MonkeyCode's free model access and free server option are useful. The operator states that both exist; confirm the current token allowance and server details on the project page before you plan a cohort, because quota and hosting specifics change and this article deliberately does not pin numbers. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A free endpoint does not remove rate limits, and it does not make a cache safe. Treat it as a budget source for the measurement block and nothing more; if a cohort of twenty students runs the replay harness simultaneously, stagger the runs and expect throttling to distort p95.

Limitations and who should skip this

• The harness assumes a request/response interface; streaming responses need a different store shape and are out of scope for 75 minutes.
• If your prompts are assembled by an agent at runtime, the key must include the assembled result, not the seed instruction.
• Teams without request logging cannot build the replay set, and should spend the session instrumenting instead.
• Anyone caching across tenants should stop and get a security review before running this lab.

Closing note

The takeaway is not "add a cache". It is that cache safety is a set of testable claims, and a team can write those tests in under an hour. Run the seven cases against your own request class this week; if the harness starts cleanly against a free model endpoint, you have a reusable safety net for every prompt change that follows.