Benchmarks tell you how often a cache helps. They don't tell you when it hurts. For an agent cache, the dangerous question isn't "how many hits did you get?" — it's "did you ever serve the wrong trajectory?" One wrong replay of a write step — deactivate instead of activate, refund instead of charge — costs more than a thousand saved tokens are worth.
So before trusting any cache on agent traffic, build it an enemy test: pairs of requests that are semantically close but must never be confused.
How the pairs were built
The Pomerene adversarial eval is 18 near-opposite command pairs. Each pair shares topic, vocabulary, and structure, and differs only in the action that matters:
- "activate user 4471" vs. "deactivate user 4471" — 0.853 cosine similarity
- Same shape, different verbs: approve vs. reject, grant vs. revoke, charge vs. refund, publish vs. unpublish
The construction rule is simple and portable: take a real command from your own domain, flip exactly one load-bearing word, and keep everything else identical. If your cache can't tell the pair apart, it will eventually serve the wrong one — because real users phrase the same goal a dozen ways, and "close enough" in embedding space is not "same" in effect space.
Embeddings capture topic, not polarity. That's the whole problem in one sentence: activate and deactivate are about the same user, the same account, the same system — so they land near each other in vector space. A threshold-only cache, the kind that serves anything above a similarity cutoff, has no principled reason to refuse. On this eval, the threshold-only baseline scored 1/18 operational false positives: one pair out of eighteen where it would have served the wrong trajectory.
The gated engine scored 0/18. The difference is the write-safety gate: any trajectory containing a write replays only on an exact repeat, never on a "close enough" match, and partial drift hands control back to the agent instead of executing a subtly wrong plan.
Run it as a CI gate, not a one-off experiment
The eval ships wired as a CI gate, and that's the part worth copying even if you never touch Pomerene. A safety eval that runs once before launch decays the moment your prompts, tools, or thresholds change. One that runs on every change catches the regression when someone "tunes" the similarity threshold up for a better hit rate and quietly reopens the polarity hole.
The method transfers to any domain:
- List your dangerous verbs. Every action your agent can take that changes state — charge, refund, send, delete, grant, approve, publish. These are the words a wrong replay must never confuse.
- Build near-opposite pairs for each. Flip one verb, keep the rest. Aim for pairs your embedding model scores high — the pairs that look safe to a threshold are exactly the ones that aren't.
- Assert zero false positives, not a good score. This eval has one passing grade. A 17/18 is a cache that will serve the wrong write on your eighteenth dangerous pair.
- Re-run on every change to matching logic. Threshold, embedding model, normalization, trajectory recording — any of it moves the boundary, so the gate runs with the change.
Honest limits — including ours
Eighteen pairs is a small eval, and it tests the gate, not the whole product. It says nothing about your domain's verbs until you build your own pairs for them — the kit's eval is a template, not a verdict on your traffic. And a cache's hit rate still depends on how repetitive your goals are; if your traces show mostly unique tasks, measure first (the diagnostics endpoint, /v1/diagnostics, shows hit rates, L1/L2/miss breakdown, latency percentiles, and dollars and tokens saved for your real traffic).
One more honest note, because this post is about the gate and the gate's story changed: the write-safety gate as originally published had three holes, found via outside critique and fixed in core — merged, shipping in the next core release. An abstract-goal match is no longer treated as an exact repeat (write trajectories are refused there, and context is diffed); a trajectory with no taint records plus a context change is now a miss; and a hit never serves past a declared external_write — control hands back to the agent at that step. Still open, not shipped: entity agreement on read matches. The eval above tests the gate as designed; the fixes are what keep the design honest, and the open item is the next gate to build. Don't take any of this on faith — run the pairs.
Pomerene ships as a self-hosted Starter Kit — Indie $99 / Pro $249, one-time, source included — with the adversarial eval wired as a CI gate so you can run it on your own domain's dangerous pairs. Docs and details at pomerene.io/starter-kit.
Top comments (0)