DEV Community: Kyle Ma

How much of your agent traffic actually repeats? Measure before you cache

Kyle Ma — Sun, 25 Oct 2026 02:36:07 +0000

How much of your agent traffic actually repeats? Measure before you cache

Trajectory caching only pays when your agent keeps solving the same problems. So before adopting it — ours or anyone's — measure whether your traffic actually repeats. This post is the checklist we'd want an engineer to run first, including the cases where the honest answer is "don't bother."

Start with the uncomfortable case

Pomerene saves little when every task is unique. If your agents do one-off bug fixing, where each ticket is a novel investigation, or bespoke per-customer work that never recurs in recognizable form, there is nothing stable to replay. A cache over unique traffic is a lookup table that always misses: you've added infrastructure, storage, and a matching step to every request in exchange for a hit rate near zero. We'd rather say that plainly than sell you a kit that can't help.

The workloads where trajectory caching earns its keep are the opposite shape: support agents fielding the same intents, booking flows, FAQ agents, scheduled jobs — anywhere the goal recurs even when the wording doesn't.

What to measure

You don't need new tooling to get a first read. Your existing traces and logs are enough for four numbers:

1. Goal repeat rate. Group completed runs by the goal they were trying to achieve — not the literal prompt, the intent. What fraction of this week's runs had a goal that also appeared last week? This is your ceiling. No cache can hit more often than your goals repeat, and if the repeat rate is under roughly a fifth of traffic, the economics get thin fast.

2. Paraphrase clustering. Raw string deduplication will lie to you. "Where's my refund?" and "I need an update on the money you owe me" are the same goal with zero tokens in common. Cluster requests by embedding similarity and look at cluster sizes: a long tail of singletons means unique work; a few fat clusters mean a small set of goals dominates your traffic, which is exactly the shape a trajectory cache exploits.

3. Cost per run. Sum tokens across all steps of a run, not just the final call — agent costs hide in the intermediate tool-use and reasoning steps. Multiply by your repeat rate and you have an upper bound on savings, in dollars, before you build anything.

4. Latency distribution. Record end-to-end run latency, p50 and p95. Long multi-step runs are where replay helps most: a fully replayed trajectory skips the entire step sequence, so the longer and more repetitive your runs, the bigger the latency win per hit.

What Pomerene reports once it's running

If you do deploy it, Pomerene ships diagnostics views so you can verify the cache is doing what the back-of-envelope math promised:

Hit rates per tenant and global — whether the repeat rate you measured in traces shows up as actual hits, and whether one tenant's traffic behaves differently from another's.
L1 / L2 / miss breakdown — full replays, partial replays (where your agent resumed at the step the cached plan stopped matching), and clean misses. A healthy deployment isn't all L1; a meaningful L2 share means the cache is absorbing the common prefix of your work even when goals partially diverge.
Latency percentiles on the hit path versus live runs.
Dollars and tokens saved, with accounting that matches how replay actually works: an L1 hit counts all of the trajectory's tokens (the whole run was skipped), while an L2 hit counts only the replayed fraction. No double-counting the steps your agent still executed.

Quality is monitored, not assumed. A /v1/feedback loop lets outcomes feed back into the cache: entries whose replay quality drops get evicted rather than served forever on the strength of one good run.

The honest limits

Two constraints worth knowing before you design around it. Pomerene currently runs as a single-writer instance with an in-memory vector index — simple and fast, but it's one process owning the cache, not a distributed store. And because stored trajectories are indexed by embedding, changing the embedding model invalidates stored vectors: swap models and the index has to be rebuilt rather than migrated in place. Neither is hidden in fine print; both should factor into whether this fits your deployment today.

The decision rule

Measure goal repeat rate and paraphrase clustering first. If a third or more of your traffic is goals you've already solved, trajectory caching will very likely pay for itself in tokens and latency — and you'll be able to prove it from the diagnostics within a week. If your traffic is genuinely one-of-a-kind work, skip the cache and spend the effort elsewhere. Either way, you'll have made the call from your own numbers instead of a vendor's benchmark.

Pomerene's self-hosted Starter Kit — Indie $99 / Pro $249, one-time, source included — comes with the diagnostics views described above, so the measurement doesn't stop at adoption. See pomerene.io/starter-kit.

Your semantic cache will happily replay the wrong command

Kyle Ma — Fri, 23 Oct 2026 22:50:48 +0000

Your semantic cache will happily replay the wrong command

Semantic caching works well for LLM responses because a near-match is usually good enough. Ask a slightly different question, get the cached answer, save a call. The failure mode is a mediocre answer.

Trajectory caching — caching the sequence of tool calls an agent makes, not just its final text — has a different failure mode. The cache doesn't store prose. It stores actions. And a near-match on actions can be the opposite operation.

Concrete example. An agent runs the goal "deactivate user 4471" and the trajectory gets cached. Later, a request comes in: "activate user 4471". With text-embedding-3-small, those two goals score 0.853 cosine similarity. A plain semantic cache running a 0.85 threshold — a common default — scores that as a hit and replays the deactivation trajectory in response to an activation request.

The embedding didn't fail. The two sentences are on the same topic, mention the same entity, and have the same structure. Embeddings are good at topic. The token that flips the operation — the polarity — barely moves the vector. If your cache decides hits on vector distance alone, it will eventually serve a write it shouldn't.

The asymmetry that matters

For agent trajectories there are two directions a fuzzy match can go wrong, and they are not equivalent:

A cached write trajectory fuzzy-matches a different request. This is the dangerous direction. The system serves actions the requester never asked for.
A cached read trajectory fuzzy-matches a write request. This returns a wrong answer — but not a destructive one, because in Pomerene cached steps are never re-executed. A hit returns the recorded results of the stored steps. Nothing runs. No tool is called. The user gets stale read data presented as an answer, which is a quality bug, not an unwanted write.

We call the first case an operational false positive: a fuzzy hit that would serve a trajectory whose effect differs from what was requested. That's the metric a trajectory cache should be designed around — not raw hit rate.

The gate: writes are exact-only

Pomerene is a deterministic trajectory cache, and its matching rule is built around that asymmetry:

Every stored tool step declares its side effect. Each step in a trajectory is tagged read, replay_safe_write, or external_write. If a step doesn't declare one, the store API rejects the trajectory. You can't opt out of the classification by forgetting to annotate; unclassified trajectories don't get stored.
Any trajectory containing a write step is served only on an exact goal+context repeat. Fuzzy (embedding-based) matching still applies to read-only trajectories, where a near-match costs you a suboptimal answer. The moment a trajectory writes, it leaves the fuzzy path entirely. A paraphrase of a write goal is a miss, full stop — the agent runs live.

Illustrative store call:

curl -X POST localhost:8080/v1/store \
  -H 'content-type: application/json' \
  -d '{
    "goal": "deactivate user 4471",
    "steps": [
      { "tool": "users.get",        "side_effect": "read",
        "args": { "id": 4471 } },
      { "tool": "users.deactivate", "side_effect": "external_write",
        "args": { "id": 4471 } }
    ]
  }'
# This trajectory now serves ONLY on an exact repeat of goal + context.
# "activate user 4471" — 0.853 similar — is a miss, and runs live.

What the eval shows

We ran an adversarial eval of 18 near-opposite command pairs — activate/deactivate-style goals designed to sit close together in embedding space — using text-embedding-3-small at a 0.85 threshold:

Configuration	Operational false positives
Threshold-only semantic cache	1 / 18
Pomerene (side-effect gate)	0 / 18

One in eighteen sounds small until you price it: that's one wrong write served, silently, out of every eighteen similar command pairs that pass through the cache. Writes are exactly the operations you can't undo with an apology.

The honest tradeoff

The gate is not free, and we won't pretend otherwise. On these same deliberately short commands, the gated cache's paraphrase recall was 1/18 — low. That's partly the test being adversarial (two-to-four-word commands carry almost no signal beyond the verb, and the verb is exactly what the gate refuses to guess at), but the direction of the tradeoff is real and deliberate:

A miss costs you one live agent run — the thing you were trying to cache in the first place.
A wrong write costs you an incorrect mutation that may not surface until much later.

Pomerene is tuned to pay the first cost and never the second. If your workload is read-heavy research and summarization, a looser cache may serve you better. If your agents mutate state, the recall you lose to the gate is the price of being able to trust a hit.

Where it sits

Pomerene wraps your agent function, not your model client, and never touches your prompts. On the hit path there is no LLM involved at all — matching is hashing plus vector geometry. Three outcomes per lookup:

L1 — full replay: the whole trajectory serves from cache. A 14-step trajectory replays in 47ms vs. 3.9s live.
L2 — partial replay: a prefix matches; when the trajectory drifts, control hands back to the agent to finish live.
Miss: the agent runs normally, and the new trajectory can be stored.

Because it wraps the agent rather than a framework's internals, it works with LangGraph, Vercel AI SDK, Mastra, or a custom loop. Quickstart is Node 20+ and Postgres; npm run setup gets you running in about five minutes.

Try it

Pomerene ships as a self-hosted Starter Kit — Indie $99 / Pro $249, one-time, source included. If your agents perform writes, run the near-opposite test against your current cache before you trust it in production.

pomerene.io/starter-kit