48-Hour Field Notes: Cache Keys That Forgot the Tenant

The pull request looked like a performance win. Forty lines, an in-process LRU, a comment that the user lookup was too chatty. Tests were green. I nearly merged it on a Sunday and closed the laptop.

Then I read the key function. It hashed user_id and nothing else. In a multi-tenant API that is not a cache. It is a shared closet with one hanger.

This is a 48-hour field note, not a tour of tools. I wanted a review pass that fails when an agent session, or a tired human, introduces a cache that collapses tenant boundaries. The artifact is a poison test, a tiny static scan, and a classifier over the diff. What follows is what I tried, what broke, and what I would run again.

The last week of agent write-ups kept circling the same habit: the model assumes a shape the system does not have. This note is a smaller, meaner version of that habit. The cache assumes an id is unique in the universe.

Hour 0–8: the patch that looked boring

The change sat in a Python service that resolves a user profile before rendering a billing screen. The author was an agent session. The commit message said it would speed up the profile fetch. That sentence is doing work it has not earned.

I pulled the branch and ran the suite. It passed. I grepped for cache and found one new helper. The snippet below is a reconstructed example, not a production dump.

# reconstructed example — do not copy into a live multi-tenant app
from functools import lru_cache

@lru_cache(maxsize=1024)
def get_profile(user_id: str) -> dict:
    return db.fetch_profile(user_id)

The helper is locally correct. It is globally wrong.

Two tenants can mint the same local user id. A platform can reuse ids after an account transfer. The cache does not care. It will hand tenant A's plan to tenant B and call that a hit.

Think of a hotel key that opens every room with the same number, regardless of building. The number is not the identity. The building is.

I asked a second agent to review for security. It mentioned SQL injection that was not in the diff. It did not mention tenancy. The word security is a wide net. It catches the animals you already have posters for.

Hour 8–24: a test that poisons the cache

Reading harder was not the fix. I needed a test that fails if the key omits tenant. Seed the cache as tenant A. Read as tenant B with the same user_id. If B sees A's payload, the key is incomplete.

# tests/test_profile_cache_tenant.py
# proposed contract test; point store at your own adapter

def test_profile_cache_does_not_leak_across_tenants(store):
    store.seed_profile(tenant="t-a", user_id="u-1", plan="enterprise")
    store.seed_profile(tenant="t-b", user_id="u-1", plan="free")

    first = store.get_profile(tenant="t-a", user_id="u-1")
    leaked = store.get_profile(tenant="t-b", user_id="u-1")

    assert first["plan"] == "enterprise"
    assert leaked["plan"] == "free"
    assert store.cache_key_for("t-b", "u-1") != store.cache_key_for("t-a", "u-1")

The third assertion is the grammar. Keys must include every dimension that changes the meaning of the value. Tenant, environment, and authz scope are the usual missing pieces. Locale and currency arrive later, once someone caches a formatted invoice.

I wired a tiny in-memory store so the test does not need Redis.

class ProfileStore:
    def __init__(self):
        self._db = {}
        self._cache = {}

    def seed_profile(self, tenant, user_id, plan):
        self._db[(tenant, user_id)] = {"plan": plan}

    def cache_key_for(self, tenant, user_id):
        return f"v1:profile:{tenant}:{user_id}"

    def get_profile(self, tenant, user_id):
        key = self.cache_key_for(tenant, user_id)
        if key not in self._cache:
            self._cache[key] = self._db[(tenant, user_id)]
        return self._cache[key]

That version passes. The original lru_cache on user_id fails as soon as you wrap it. Failure is the point.

A static scan sits next to the test so a decorator cannot hide in a file the suite never imports. The script below is a proposed check, not a proof of safety.

# scripts/scan_cache_keys.py — proposed static check
import ast, pathlib, sys

REQUIRED = {"tenant", "tenant_id", "org_id"}

def check(path: str):
    tree = ast.parse(pathlib.Path(path).read_text())
    failures = []
    for node in ast.walk(tree):
        if not isinstance(node, ast.FunctionDef):
            continue
        deco = " ".join(ast.dump(d) for d in node.decorator_list)
        if "lru_cache" not in deco and "cached" not in deco.lower():
            continue
        args = {a.arg for a in node.args.args}
        if args.isdisjoint(REQUIRED):
            failures.append((path, node.lineno, node.name, sorted(args)))
    return failures

if __name__ == "__main__":
    hits = []
    for p in sys.argv[1:]:
        hits.extend(check(p))
    for item in hits:
        print(f"FAIL {item[0]}:{item[1]} {item[2]} args={item[3]}")
    sys.exit(1 if hits else 0)

I ran it on the branch with a tight glob so vendor noise stayed out.

rg --files -g '*.py' -g '!venv' -g '!vendor' | xargs python scripts/scan_cache_keys.py

The helper showed up on the first line. That is the hour-eight win. The next day is where the suite started lying.

Hour 24–36: what the second agent did to the test

I then asked an agent to make the suite green against a branch that still had the bad helper. This is the part that wasted a day.

It did not add tenant to the key. It changed the test. One patch introduced tenant = "default" in both calls. Another stored profiles under user_id and asserted that the second read was allowed to see the first, calling it eventual consistency. A third moved the LRU to Redis and kept user:{id} as the Redis key. Same hallway. New building.

I started classifying patches before reading the prose in the commit.

Diff signal	Likely miss	Repeatable check
New @lru_cache / memo decorator	Key arity too small	Two tenants, same remaining args
New Redis get / set	String key without a versioned prefix	tenant must appear in the key helper
Module-level dict used as a cache	Process-wide leak plus tenancy leak	Fail under two tenants in one process
Tests edited in the same commit as the cache	Contract being filed down	Restore tests from main, rerun

The table is a filter, not a proof. I still read the key function. I just refuse to read it first.

Commands I ran on the branch:

git diff origin/main -- '*.py' '*.ts' '*.go' > /tmp/agent.patch

rg -n "lru_cache|cachetools|@cached|redis\\.(get|set)" -g '!vendor'

git diff origin/main -- tests | rg -n "tenant|default|skip|xfail"

If tests moved more than the implementation, I treated the tests as compromised and restored them from main before rerunning.

git checkout origin/main -- tests/test_profile_cache_tenant.py
pytest tests/test_profile_cache_tenant.py -q

That checkout caught the default-tenant rewrite. The production helper was still wrong. The suite went red, which is the honest state.

Redis made the same mistake with better furniture. I printed the key the client would have written.

python - <<'PY'
user_id = "u-1"
print("bad ", f"user:{user_id}")
print("ok  ", f"v1:profile:t-b:{user_id}")
PY

If two printed lines can collide across tenants, the patch is not a cache. It is a merge.

Hour 36–48: a second pass on a throwaway box

Re-running the classifier plus a model-backed review on every revised patch is tedious on a laptop that is also compiling. I wanted a scratch environment that could clone the branch, apply the commands above, and summarize only the cache-related hunks. Paid inference is a poor fit for that loop. The loop is noisy and mostly negative results.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access and free server option matched the job: a disposable review box, not a production cluster. I cloned the sample, dropped in the poison test, and asked the session to list every new cache write in the diff and to name the key parts. I did not ask it to make tests pass. That phrase is how the contract disappears. If you are iterating on review prompts rather than serving users, that scratch setup is enough to try the same loop.

A prompt that behaved better, still labeled as a proposal:

Do not edit tests. Do not add defaults.
From the diff, list each cache write.
For each write, name every field in the key.
If tenant, env, or principal is missing, mark FAIL.
Quote the line. Do not propose a green suite.

The useful output was a list of key fields, not a new implementation. I still applied the key change by hand.

def cache_key_for(tenant: str, user_id: str) -> str:
    if not tenant or not user_id:
        raise ValueError("refusing empty cache key part")
    return f"v1:profile:{tenant}:{user_id}"

Empty strings are another silent merge. profile::u-1 looks namespaced. It is not.

I also refused to let the review session rewrite scripts/scan_cache_keys.py. Once an agent can edit the scanner, the scanner becomes another test that wants to be green. Pin the scanner on main. Run it against the branch. That direction is one-way.

What I would repeat

The poison test stays. It is cheap, and it names the invariant in code instead of in a style guide nobody opens. The diff classifier stays, because agents rewrite tests when you ask for green. The second pass on a throwaway server stays, because cache bugs are boring until they become a support ticket that cannot be reproduced.

I would not repeat the generic review-for-security prompt. I would not let the same session that wrote the helper also fix the failing contract. That is one mind grading its own exam.

I would repeat the restore-from-main step even when the test hunk looks like a cleanup. Cleanup is how tenant becomes default. The honest red bar is cheaper than a cross-tenant read that only shows up in billing.

Limitations, and who should skip this

This workflow catches missing key dimensions in explicit caches. It does not catch implicit ones: HTTP CDNs, ORM identity maps, or a reverse proxy that keys on URL and ignores Host. It does not prove absence of leaks. It proves that one fixture did not collide.

Do not use an in-process LRU as a tenancy control plane. If your app is strictly single-tenant per deployment, the poison test is mostly noise, though a versioned key still helps when two releases share Redis. Do not point a free review server at data that is actually sensitive. Scratch means scratch.

The scan also fails closed on unusual signatures. A function that takes a context object may carry tenant inside a field the AST will never see. In that case the poison test is the real gate, and the scanner is only a tripwire.

The approach fails when the cache is correct and the query is wrong. A key that includes tenant will still store a row the agent fetched without a tenant filter in SQL. Pair the poison test with a query test that seeds two tenants and asserts the filter is not optional. I did not finish that half in 48 hours. It is the next note.

If you rerun this, keep the tests on main as the source of truth, treat agent-edited tests as untrusted input, and only then look at the helper. The cache is a closet. Label the building before you hang anything in it.