Two people asked a context compressor two completely different questions. It gave them the same answer. Not a similar answer — byte for byte the same 544 characters.
Here's what that looked like:
query="Fix the IntegrityError on commit" level=L0 -> 159 tok cache_hit=False
query="Explain the tax rounding TODO in compute_tax" level=L3 -> 159 tok cache_hit=True
identical output: yes (544 chars both)
Different question. Different compression level. Same 544 characters, served from cache.
Finding it
I wasn't looking for this. I was auditing something else entirely — measuring how much meaning a context compressor loses, not how fast it runs. My harness feeds the same corpus through the compressor with different queries and checks which critical substrings survive: file paths, error types, line numbers, identifiers.
I noticed two rows in my results table were identical. Same token count, same output. My first assumption was that my own harness had a bug — that I was passing the same query twice and hadn't noticed. So I changed the second query to something with no words in common with the first, and bumped the compression level from L0 to L3, which should change the output dramatically on its own.
Same 544 characters.
That was the moment it stopped being my bug.
The cause
One line:
sid = content_hash(content)
That sid was doing two jobs. It was the shadow ID — the handle used to refer to a stored document. And it was also the cache key.
As a shadow ID it's correct: the same content should get the same handle. As a cache key it's wrong, because the output of compress() doesn't depend only on the content. It depends on the content and the query and the compression level. Two of those three inputs were simply not part of the key.
So the first caller warmed the cache for a piece of content, and everyone who touched that same content afterwards got the first caller's answer — regardless of what they actually asked for.
Why this is worse than a stale cache
A stale cache gives you an old answer to your question. You can reason about that. You add a TTL, you invalidate on write, you move on.
This gives you a fresh, correct answer to somebody else's question, and it does it silently. There's no error, no warning, no log line. The output is well-formed. It's the right shape, the right length, the right format. It just answers a question you didn't ask.
And the failure gets worse the better the system works. The more people share a document — a popular file, a common stack trace, a template — the more of them collide on the same key. The bug is quietest exactly where the traffic is heaviest.
That's what makes this class of bug nasty: it doesn't look like a bug. It looks like a cache doing its job.
The fix
Key the cache on everything the output depends on, and keep the shadow ID separate:
sid = content_hash(content) # stable handle for the content
cache_key = hash((sid, query, level)) # what the answer actually depends on
The shadow ID keeps its property — same content, same handle. The cache key now changes when the intent changes, which is what makes a cache hit mean something.
What I took away from it
When one variable does two jobs, one of the jobs is probably wrong. The name sid described the first job. Nothing in the code said it was also the cache key — it just was, by virtue of being passed to the cache. The bug lived in the gap between what the variable was named and what it was used for.
Ask whether your cache key covers every input to the function. Not "does this look unique" — actually list the parameters and check them off. compress(content, query, level) has three inputs; the key had one. Written down like that, it's obvious. Nobody writes it down.
And test with inputs that share nothing. My original two queries were both about Python errors, and if the output had been merely similar I'd have shrugged and moved on. It was changing the second query to something with zero overlap — and a different level — that turned a shrug into a bug report. When you suspect two things are being conflated, make them as different as you possibly can and see if the system still can't tell them apart.
The fix is open as a PR. The audit tool that surfaced it is here — it measures what compression costs you, not just what it saves.
Top comments (0)