DEV Community: Kyle Ma

Your agent bill isn't decided by the model price. It's decided by your repeat rate.

Kyle Ma — Sun, 15 Nov 2026 04:57:27 +0000

Your agent bill isn't decided by the model price. It's decided by your repeat rate.

A 20-step agent task re-sends its context at every step. That is the whole cost story, and it is easy to miss, because the price you shop for — dollars per million tokens — is the smallest term in the bill. The terms that actually decide it are in your own traces: how often the same goal repeats, how long the gaps between repeats are, and what a finished task costs you end to end.

An external worked example makes the shape concrete. Sourceit.com.sg's cost breakdown of a 20-step task on Sonnet 5.5 puts the re-sent input at 770k tokens, $2.14 per task without caching versus $0.97 at an 80% cache rate — and at 1,000 tasks a day for 30 days, $64,200 versus $29,088. Their retry-loop number is the one to sit with: one agent stuck in a loop for 12 hours costs about $318, ten agents about $3,180. Those are their numbers, from their model and their workload — but the structure generalizes: the bill is steps × re-sent context × repeats, and the multiplier that nobody budgets for is the loop.

Spheron's roundup of the same problem cites Gartner 2026 — agentic AI consumes 5–30x more tokens per task than a chatbot exchange — and a Stanford Digital Economy Lab finding that identical SWE-bench runs can differ up to 30x in token cost, because trajectories are stochastic. Tool calls add tokens twice, once for the schema and once for the result. Their conclusion is the same one this post is built on: the highest-leverage fix attacks the re-sent context directly.

The three numbers, in order

1. Repeat rate — but measured on goals, not prompts. A repeat rate computed on exact prompt text understates the opportunity, because the same goal arrives in different wording all day. On τ-bench — the airline customer-service benchmark, dense with the rephrased-same-goal traffic that support agents actually see — Pomerene's trajectory cache served 86% of paraphrased requests from cache, with 78% of those judged correct. Read both numbers honestly: 86% is why repetitive agent workloads are worth caching, and 78% is why the safety rules below are the product, not a footnote. Your number will differ. Measure it from your traces: cluster completed runs by goal, count the clusters with more than one member, and that is your ceiling. If goal repeats across sessions are effectively zero on your corpus, a trajectory cache has almost nothing to serve there — and the honest answer is to not buy one.

2. The gap distribution — because a repeat after the data changed is a stale hit, not a saved run. A repeat rate without a time dimension is a vanity metric. Bucket your requests by gap length first, then price each bucket separately: a repeat inside the TTL is a saving, a repeat after the underlying data changed is a wrong answer served fast. Which TTL wins — 5-minute or 1-hour — falls out of the gap histogram, not the average task cost. The same logic applies to the cache-write side: a small fraction of requests can write a large fraction of the cache-write tokens, so price the writes per bucket too, not per task.

3. Cost per finished task — not per token, not per request. Cost per token flatters any change that makes each call cheaper while the agent takes more turns to finish. Cost per request hides the retry loops. The decision-grade number is what a completed task costs, including the runs that failed and retried. A cache that never hits is worse than no cache at all — it charges you the lookup and validation on every call and saves nothing — so the first test of any caching change is whether cost per finished task actually falls.

Where trajectory replay fits in that math

Prefix caching and trajectory caching attack different terms, and they compose. Prefix caching reuses the re-sent context and still generates — it makes each completion cheaper. A trajectory cache replays the recorded plan instead of re-deriving it: a 14-step audit trajectory replays in 47ms versus 3.9s live, with zero LLM invocation on the hit path. The saving lands on the whole run, not on one call's input tokens — which is exactly the term the 5–30x multiplier lives in.

The match has three outcomes, all deterministic — hashing plus vector geometry computed in TypeScript, no LLM call in the match decision, so the same input makes the same cache decision every time. L1 is a full replay. L2 replays the matching prefix and hands control back to your agent at the exact step where the plan stops matching. A miss runs live and gets stored. That L2 handoff is what makes the math safe to apply to real traffic: partial drift hands control back to the agent, never a subtly wrong execution.

The safety term in the equation

A wrong hit doesn't show up in your token bill. It shows up as a wrong answer, a duplicated charge, a state change that never should have happened — and no cost model that omits it is honest. So the write rule is the whole product: steps that write are never fuzzy-matched against a new request. A cached write step replays only on an exact goal-plus-context repeat; anything ambiguous becomes a miss or an L2 handoff. On an adversarial eval of polarity-flipped pairs (requests that are semantically close but must not be confused, like activate vs. deactivate at 0.853 cosine similarity), the gated engine produced 0 false positives across 18 pairs.

One honest note, because the gate's story changed after the first post about it: the write-safety gate as originally published had three holes, found via outside critique and fixed in core — merged, shipping in the next core release. An abstract-goal match is no longer treated as an exact repeat (write trajectories are refused there, and context is diffed); a trajectory with no taint records plus a context change is now a miss; and a hit never serves past a declared external_write — control hands back to the agent at that step. Still open, not shipped: entity agreement on read matches. Any cost model you build should treat those as the current rules, and the open item as the next gate to build.

Do the math before you buy the cache

If your traces show the same goals arriving in different wording, you're paying the model to rediscover plans it already derived. If they don't, no cache — trajectory, prefix, or otherwise — will save you money, and the kit will tell you that before you commit: the built-in diagnostics (/v1/diagnostics) show hit rates, the L1/L2/miss breakdown, latency percentiles, and dollars and tokens saved for your real traffic. Run the three numbers first — goal repeat rate, gap histogram, cost per finished task — and let them decide whether caching is a cost lever or a distraction for your workload.

Pomerene ships as a self-hosted Starter Kit — Indie $99 / Pro $249, one-time, source included — so the cache runs in your infrastructure, next to your agents. Quickstart is about 5 minutes (Node 20+, Docker or Postgres 15+). Docs and details at pomerene.io/starter-kit.

Your Vercel AI SDK agent already knows the answer. Stop making it re-derive it.

Kyle Ma — Sun, 08 Nov 2026 05:00:49 +0000

Your Vercel AI SDK agent already knows the answer. Stop making it re-derive it.

Your Vercel AI SDK agent just spent 3.9 seconds and a full tool loop answering a goal it answered yesterday. Different user, slightly different wording, same intent — and the agent re-derived the whole plan from scratch: same tool choices, same order, same result. The model call isn't the expensive part. The re-derivation is.

This is the loop tax, and it hits every framework the same way. Vercel AI SDK's generateText with tools, Mastra's agent runs, a hand-rolled loop — they all re-run the same goals all day, because nothing in the stack notices that the trajectory repeated. Prompt caching at the model layer saves you per-call. It doesn't notice that the plan repeated.

Pomerene is a deterministic trajectory cache for agentic workflows, and the integration model is deliberately boring: it wraps your agent function, not your model client. Your prompts are never touched, rewritten, or intercepted. That matters for Vercel AI SDK and Mastra, because the cache doesn't need to know anything about your framework's internals — only that it can record a run and replay the recorded steps when a matching goal arrives.

The integration: one wrap, or two HTTP calls

There are two integration shapes, and both are framework-proof — the same code works whether your agent runs on Vercel AI SDK, Mastra, LangGraph, or a hand-rolled loop, because the unit being cached is the trajectory, not the framework object.

One wrap:

import { withCache } from "@pomerene/core"; // from the self-hosted Starter Kit

const cachedAgent = withCache(myAgent, {
  // policy: what may be fuzzy-matched, what may never be
  writeSteps: "never-fuzzy",   // steps that mutate state are never fuzzy-matched
  drift: "handoff",            // partial drift hands control back to the agent
});

const result = await cachedAgent.run("refund the extra charge on my last order");

Or two HTTP calls, in any language:

POST /v1/lookup { goal, context }   # before your agent runs
POST /v1/store   { goal, trajectory } # after it finishes

L1 returns the cached outcome. L2 resumes your agent mid-trajectory. A miss runs live and gets stored. That's the whole integration — and it works the same way in a Vercel AI SDK tool loop as in a Mastra agent, because in both cases the agent is a function: goal in, result out, plus a recorded trajectory in between.

What a hit actually looks like

Three outcomes, all deterministic — hashing plus vector geometry computed in TypeScript, no LLM call in the match decision, so the same input makes the same cache decision every time:

L1 — full replay. The incoming goal matches a cached trajectory end to end. The stored plan replays: a 14-step audit trajectory replays in 47ms versus 3.9s live, with zero LLM invocation on the hit path.
L2 — partial replay. The cached plan matches up to a point, then diverges. Pomerene replays the matching prefix and hands control back to your agent at the exact step where the plan stops matching. The agent only does the work that's actually new — the tool loop resumes mid-trajectory instead of restarting.
Miss. No usable match — the request runs through your agent normally, and the completed trajectory is recorded for future hits.

On τ-bench — the airline customer-service benchmark, dense with the rephrased-same-goal traffic that support agents actually see — the engine served 86% of paraphrased requests from cache, with 78% of those judged correct. Read both numbers honestly: 86% is why repetitive agent workloads are worth caching, and 78% is why the write-safety rules below are the product, not a footnote.

The part that has to be true before you cache a write

A Vercel AI SDK agent with tools can charge a card, send a message, mutate a database. A Mastra agent the same. So the write rule is the whole product: steps that write are never fuzzy-matched against a new request. A cached write step replays only on an exact goal-plus-context repeat; anything ambiguous becomes a miss or an L2 handoff. Partial drift produces a handoff to your agent, never a subtly wrong execution. On an adversarial eval of polarity-flipped pairs (requests that are semantically close but must not be confused, like activate vs. deactivate at 0.853 cosine similarity), the gated engine produced 0 false positives across 18 pairs.

One honest note, because the gate's story changed after the first post about it: the write-safety gate as originally published had three holes, found via outside critique and fixed in core — merged, shipping in the next core release. An abstract-goal match is no longer treated as an exact repeat (write trajectories are refused there, and context is diffed); a trajectory with no taint records plus a context change is now a miss; and a hit never serves past a declared external_write — control hands back to the agent at that step. Still open, not shipped: entity agreement on read matches. The numbers above describe the gate as designed; the fixes are what keep the design honest, and the open item is the next gate to build.

Measure first, then wrap

If your agent's traffic is mostly unique, one-off tasks, this buys you less — and the kit will tell you that before you commit. The built-in diagnostics (/v1/diagnostics) show hit rates, the L1/L2/miss breakdown, latency percentiles, and dollars and tokens saved for your real traffic. If your traces show the same goals arriving in different wording, you're paying the model to rediscover plans it already derived. A wrapper turns that rediscovery into a replay — in your Vercel AI SDK loop, your Mastra agent, or wherever the agent lives.

How to build an adversarial eval for your agent cache (and why 18 pairs matter more than a benchmark)

Kyle Ma — Sun, 01 Nov 2026 02:59:38 +0000

Benchmarks tell you how often a cache helps. They don't tell you when it hurts. For an agent cache, the dangerous question isn't "how many hits did you get?" — it's "did you ever serve the wrong trajectory?" One wrong replay of a write step — deactivate instead of activate, refund instead of charge — costs more than a thousand saved tokens are worth.

So before trusting any cache on agent traffic, build it an enemy test: pairs of requests that are semantically close but must never be confused.

How the pairs were built

The Pomerene adversarial eval is 18 near-opposite command pairs. Each pair shares topic, vocabulary, and structure, and differs only in the action that matters:

"activate user 4471" vs. "deactivate user 4471" — 0.853 cosine similarity
Same shape, different verbs: approve vs. reject, grant vs. revoke, charge vs. refund, publish vs. unpublish

The construction rule is simple and portable: take a real command from your own domain, flip exactly one load-bearing word, and keep everything else identical. If your cache can't tell the pair apart, it will eventually serve the wrong one — because real users phrase the same goal a dozen ways, and "close enough" in embedding space is not "same" in effect space.

Embeddings capture topic, not polarity. That's the whole problem in one sentence: activate and deactivate are about the same user, the same account, the same system — so they land near each other in vector space. A threshold-only cache, the kind that serves anything above a similarity cutoff, has no principled reason to refuse. On this eval, the threshold-only baseline scored 1/18 operational false positives: one pair out of eighteen where it would have served the wrong trajectory.

The gated engine scored 0/18. The difference is the write-safety gate: any trajectory containing a write replays only on an exact repeat, never on a "close enough" match, and partial drift hands control back to the agent instead of executing a subtly wrong plan.

Run it as a CI gate, not a one-off experiment

The eval ships wired as a CI gate, and that's the part worth copying even if you never touch Pomerene. A safety eval that runs once before launch decays the moment your prompts, tools, or thresholds change. One that runs on every change catches the regression when someone "tunes" the similarity threshold up for a better hit rate and quietly reopens the polarity hole.

The method transfers to any domain:

List your dangerous verbs. Every action your agent can take that changes state — charge, refund, send, delete, grant, approve, publish. These are the words a wrong replay must never confuse.
Build near-opposite pairs for each. Flip one verb, keep the rest. Aim for pairs your embedding model scores high — the pairs that look safe to a threshold are exactly the ones that aren't.
Assert zero false positives, not a good score. This eval has one passing grade. A 17/18 is a cache that will serve the wrong write on your eighteenth dangerous pair.
Re-run on every change to matching logic. Threshold, embedding model, normalization, trajectory recording — any of it moves the boundary, so the gate runs with the change.

Honest limits — including ours

Eighteen pairs is a small eval, and it tests the gate, not the whole product. It says nothing about your domain's verbs until you build your own pairs for them — the kit's eval is a template, not a verdict on your traffic. And a cache's hit rate still depends on how repetitive your goals are; if your traces show mostly unique tasks, measure first (the diagnostics endpoint, /v1/diagnostics, shows hit rates, L1/L2/miss breakdown, latency percentiles, and dollars and tokens saved for your real traffic).

One more honest note, because this post is about the gate and the gate's story changed: the write-safety gate as originally published had three holes, found via outside critique and fixed in core — merged, shipping in the next core release. An abstract-goal match is no longer treated as an exact repeat (write trajectories are refused there, and context is diffed); a trajectory with no taint records plus a context change is now a miss; and a hit never serves past a declared external_write — control hands back to the agent at that step. Still open, not shipped: entity agreement on read matches. The eval above tests the gate as designed; the fixes are what keep the design honest, and the open item is the next gate to build. Don't take any of this on faith — run the pairs.

Pomerene ships as a self-hosted Starter Kit — Indie $99 / Pro $249, one-time, source included — with the adversarial eval wired as a CI gate so you can run it on your own domain's dangerous pairs. Docs and details at pomerene.io/starter-kit.

How much of your agent traffic actually repeats? Measure before you cache

Kyle Ma — Sun, 25 Oct 2026 02:36:07 +0000

How much of your agent traffic actually repeats? Measure before you cache

Trajectory caching only pays when your agent keeps solving the same problems. So before adopting it — ours or anyone's — measure whether your traffic actually repeats. This post is the checklist we'd want an engineer to run first, including the cases where the honest answer is "don't bother."

Start with the uncomfortable case

Pomerene saves little when every task is unique. If your agents do one-off bug fixing, where each ticket is a novel investigation, or bespoke per-customer work that never recurs in recognizable form, there is nothing stable to replay. A cache over unique traffic is a lookup table that always misses: you've added infrastructure, storage, and a matching step to every request in exchange for a hit rate near zero. We'd rather say that plainly than sell you a kit that can't help.

The workloads where trajectory caching earns its keep are the opposite shape: support agents fielding the same intents, booking flows, FAQ agents, scheduled jobs — anywhere the goal recurs even when the wording doesn't.

What to measure

You don't need new tooling to get a first read. Your existing traces and logs are enough for four numbers:

1. Goal repeat rate. Group completed runs by the goal they were trying to achieve — not the literal prompt, the intent. What fraction of this week's runs had a goal that also appeared last week? This is your ceiling. No cache can hit more often than your goals repeat, and if the repeat rate is under roughly a fifth of traffic, the economics get thin fast.

2. Paraphrase clustering. Raw string deduplication will lie to you. "Where's my refund?" and "I need an update on the money you owe me" are the same goal with zero tokens in common. Cluster requests by embedding similarity and look at cluster sizes: a long tail of singletons means unique work; a few fat clusters mean a small set of goals dominates your traffic, which is exactly the shape a trajectory cache exploits.

3. Cost per run. Sum tokens across all steps of a run, not just the final call — agent costs hide in the intermediate tool-use and reasoning steps. Multiply by your repeat rate and you have an upper bound on savings, in dollars, before you build anything.

4. Latency distribution. Record end-to-end run latency, p50 and p95. Long multi-step runs are where replay helps most: a fully replayed trajectory skips the entire step sequence, so the longer and more repetitive your runs, the bigger the latency win per hit.

What Pomerene reports once it's running

If you do deploy it, Pomerene ships diagnostics views so you can verify the cache is doing what the back-of-envelope math promised:

Hit rates per tenant and global — whether the repeat rate you measured in traces shows up as actual hits, and whether one tenant's traffic behaves differently from another's.
L1 / L2 / miss breakdown — full replays, partial replays (where your agent resumed at the step the cached plan stopped matching), and clean misses. A healthy deployment isn't all L1; a meaningful L2 share means the cache is absorbing the common prefix of your work even when goals partially diverge.
Latency percentiles on the hit path versus live runs.
Dollars and tokens saved, with accounting that matches how replay actually works: an L1 hit counts all of the trajectory's tokens (the whole run was skipped), while an L2 hit counts only the replayed fraction. No double-counting the steps your agent still executed.

Quality is monitored, not assumed. A /v1/feedback loop lets outcomes feed back into the cache: entries whose replay quality drops get evicted rather than served forever on the strength of one good run.

The honest limits

Two constraints worth knowing before you design around it. Pomerene currently runs as a single-writer instance with an in-memory vector index — simple and fast, but it's one process owning the cache, not a distributed store. And because stored trajectories are indexed by embedding, changing the embedding model invalidates stored vectors: swap models and the index has to be rebuilt rather than migrated in place. Neither is hidden in fine print; both should factor into whether this fits your deployment today.

The decision rule

Measure goal repeat rate and paraphrase clustering first. If a third or more of your traffic is goals you've already solved, trajectory caching will very likely pay for itself in tokens and latency — and you'll be able to prove it from the diagnostics within a week. If your traffic is genuinely one-of-a-kind work, skip the cache and spend the effort elsewhere. Either way, you'll have made the call from your own numbers instead of a vendor's benchmark.

Pomerene's self-hosted Starter Kit — Indie $99 / Pro $249, one-time, source included — comes with the diagnostics views described above, so the measurement doesn't stop at adoption. See pomerene.io/starter-kit.

Your semantic cache will happily replay the wrong command

Kyle Ma — Fri, 23 Oct 2026 22:50:48 +0000

Your semantic cache will happily replay the wrong command

Semantic caching works well for LLM responses because a near-match is usually good enough. Ask a slightly different question, get the cached answer, save a call. The failure mode is a mediocre answer.

Trajectory caching — caching the sequence of tool calls an agent makes, not just its final text — has a different failure mode. The cache doesn't store prose. It stores actions. And a near-match on actions can be the opposite operation.

Concrete example. An agent runs the goal "deactivate user 4471" and the trajectory gets cached. Later, a request comes in: "activate user 4471". With text-embedding-3-small, those two goals score 0.853 cosine similarity. A plain semantic cache running a 0.85 threshold — a common default — scores that as a hit and replays the deactivation trajectory in response to an activation request.

The embedding didn't fail. The two sentences are on the same topic, mention the same entity, and have the same structure. Embeddings are good at topic. The token that flips the operation — the polarity — barely moves the vector. If your cache decides hits on vector distance alone, it will eventually serve a write it shouldn't.

The asymmetry that matters

For agent trajectories there are two directions a fuzzy match can go wrong, and they are not equivalent:

A cached write trajectory fuzzy-matches a different request. This is the dangerous direction. The system serves actions the requester never asked for.
A cached read trajectory fuzzy-matches a write request. This returns a wrong answer — but not a destructive one, because in Pomerene cached steps are never re-executed. A hit returns the recorded results of the stored steps. Nothing runs. No tool is called. The user gets stale read data presented as an answer, which is a quality bug, not an unwanted write.

We call the first case an operational false positive: a fuzzy hit that would serve a trajectory whose effect differs from what was requested. That's the metric a trajectory cache should be designed around — not raw hit rate.

The gate: writes are exact-only

Pomerene is a deterministic trajectory cache, and its matching rule is built around that asymmetry:

Every stored tool step declares its side effect. Each step in a trajectory is tagged read, replay_safe_write, or external_write. If a step doesn't declare one, the store API rejects the trajectory. You can't opt out of the classification by forgetting to annotate; unclassified trajectories don't get stored.
Any trajectory containing a write step is served only on an exact goal+context repeat. Fuzzy (embedding-based) matching still applies to read-only trajectories, where a near-match costs you a suboptimal answer. The moment a trajectory writes, it leaves the fuzzy path entirely. A paraphrase of a write goal is a miss, full stop — the agent runs live.

Illustrative store call:

curl -X POST localhost:8080/v1/store \
  -H 'content-type: application/json' \
  -d '{
    "goal": "deactivate user 4471",
    "steps": [
      { "tool": "users.get",        "side_effect": "read",
        "args": { "id": 4471 } },
      { "tool": "users.deactivate", "side_effect": "external_write",
        "args": { "id": 4471 } }
    ]
  }'
# This trajectory now serves ONLY on an exact repeat of goal + context.
# "activate user 4471" — 0.853 similar — is a miss, and runs live.

What the eval shows

We ran an adversarial eval of 18 near-opposite command pairs — activate/deactivate-style goals designed to sit close together in embedding space — using text-embedding-3-small at a 0.85 threshold:

Configuration	Operational false positives
Threshold-only semantic cache	1 / 18
Pomerene (side-effect gate)	0 / 18

One in eighteen sounds small until you price it: that's one wrong write served, silently, out of every eighteen similar command pairs that pass through the cache. Writes are exactly the operations you can't undo with an apology.

The honest tradeoff

The gate is not free, and we won't pretend otherwise. On these same deliberately short commands, the gated cache's paraphrase recall was 1/18 — low. That's partly the test being adversarial (two-to-four-word commands carry almost no signal beyond the verb, and the verb is exactly what the gate refuses to guess at), but the direction of the tradeoff is real and deliberate:

A miss costs you one live agent run — the thing you were trying to cache in the first place.
A wrong write costs you an incorrect mutation that may not surface until much later.

Pomerene is tuned to pay the first cost and never the second. If your workload is read-heavy research and summarization, a looser cache may serve you better. If your agents mutate state, the recall you lose to the gate is the price of being able to trust a hit.

Where it sits

Pomerene wraps your agent function, not your model client, and never touches your prompts. On the hit path there is no LLM involved at all — matching is hashing plus vector geometry. Three outcomes per lookup:

L1 — full replay: the whole trajectory serves from cache. A 14-step trajectory replays in 47ms vs. 3.9s live.
L2 — partial replay: a prefix matches; when the trajectory drifts, control hands back to the agent to finish live.
Miss: the agent runs normally, and the new trajectory can be stored.

Because it wraps the agent rather than a framework's internals, it works with LangGraph, Vercel AI SDK, Mastra, or a custom loop. Quickstart is Node 20+ and Postgres; npm run setup gets you running in about five minutes.

Try it

Pomerene ships as a self-hosted Starter Kit — Indie $99 / Pro $249, one-time, source included. If your agents perform writes, run the near-opposite test against your current cache before you trust it in production.

pomerene.io/starter-kit