DEV Community: Guillermo Baigorria

Prompt caching: same agent, same rules, about a quarter of the bill

Guillermo Baigorria — Mon, 02 Nov 2026 13:55:05 +0000

Prompt caching has been covered a lot. The big providers' guides explain how it works, and many posts tell how a team's hit rate jumped after one fix. We want to add a way to watch it happen: one agent's prompt, drawn the same way at every step, with the measured number in each picture. Sometimes the same idea, shown another way, is what makes it click.

To show it, we built a small recipe agent based on what happened in our own agents: a prompt that looks fine, fixed in four moves, every version sent to a real model API. Same model, same rules, not a word changed: its prompts got 73% cheaper, and the whole test cost less than a cent. At the end we show a test that keeps them from coming undone, and what the same moves did in ours.

How a prompt cache decides what you pay

Every call to a model sends the whole request again: the tools, the instructions and the conversation. A prompt cache lets the provider keep the work it already did on the start of a request, and reuse it when the next request starts exactly the same way. In the guides from OpenAI and Anthropic, reading from the cache costs a tenth of the normal input price or less, and writing to it a quarter more.

The rule behind it is short: a prompt cache reuses everything up to the first thing that changes. That is the whole game: what never changes goes first, and what changes goes last.

Four moves on a recipe agent

Our test agent writes recipes. It has two tools, find_in_pantry and save_recipe, about a thousand words of rules, and a profile for each person. The app has two plans: Pro can save recipes, free cannot. Ana is on the Pro plan and vegetarian; Leo is on the free plan. During one evening, each of them sends three messages.

A prompt that looks fine

This is the prompt many agents start with. The date goes on top, so the agent knows what day it is. Ana's details come next, so it can personalize. Her plan decides which tools she gets. And like many harnesses, it builds the whole prompt again before every call to the model.

The cache was on, and nothing came back from it. Every message wrote its whole prompt into the cache and none read it back, so the evening cost more than it would have with no cache at all.

Step 1: Move what changes on every message to the end

The date changes every minute, and it sits near the top. Whatever comes after it is new work on every message, the thousand words of rules included. Both guides say the same: what changes on every request goes at the end. So the date moves into the user message.

Each person's later messages now read almost their whole prompt from the cache, and the evening went from 0% to 65%. Anything that changes on every message goes at the end.

A prompt cache reuses everything up to the first thing that changes.

And yet Leo's first message still read nothing.

Step 2: Keep personal details out of the shared part

Ana's details change with the person. With them in the instructions, nobody else can reuse Ana's prompt. So they move into the user message, and the instructions become the same text for everyone.

The evening went down a little, to 64%, because Ana's details now travel with each of her messages. And Leo still read nothing. What belongs to one person goes after what everyone shares, but that alone was not enough: something still differed, and it came before the instructions.

Step 3: Give every plan the same tool list

The tools come before the instructions, and Leo's free plan gave him a shorter list. His request differed from Ana's at its very first line, so it could not reuse anything she had left in the cache.

The fix is to send every plan the same list. The list only describes the tools; it does not decide what Leo may use. That is decided apart: OpenAI's guide says to "restrict which tools are callable while keeping the supplied tools list stable", and the code that runs the tools refuses anything his plan does not allow. Give everyone the same menu, and decide at the table what each one may order.

Leo's first message now reused what Ana had left in the cache, and the evening reached 80%. The plan still decides what each person can do; it just stopped deciding what the prompt says.

Step 4: Tool calls almost free: append, don't rebuild

To answer Ana, the agent first checks her pantry: the model asks for find_in_pantry, the agent runs it and calls the model again with the result. If that call only adds the tool call and its result at the end, everything before them comes from the cache. OpenAI's guide puts it plainly: "Append new messages rather than rewriting earlier turns."

The call after the tool read 94% of its prompt from the cache. An agent makes many calls like this one, so they end up almost free. In the first version, the agent rebuilt the whole prompt before every call, with a fresh time on top, and the same call read nothing. A tool call should add to the request, never rebuild it.

What the four moves add up to

Together, the four moves changed what the evening cost. At a price of $1 per million input tokens, its prompts went from $1.76 to $0.48 per thousand calls, and the call after the tool cost $0.25.

None of the four moves changed a word of the rules. We ran the test on gpt-6-luna with reasoning off; for any model, multiply these costs by its price per million input tokens.

Check your own agent

Take two requests your agent sends, from two people a minute apart, and find the first place where they differ. Everything before it can come from the cache; everything after it is paid on every call. If that place is near the top, look at what sits there: a date, a name, a tool list built for one plan.

Then keep it that way with a test. This one never calls a model: it builds two requests and compares how they start.

# two people, two times: the start of their requests must be identical
ana = build_request(ANA, "6:42 PM", "What can I make tonight with chickpeas and spinach?")
leo = build_request(LEO, "6:43 PM", "A quick lunch with eggs for tomorrow?")
assert ana["tools"] == leo["tools"]                 # the same tool list
assert ana["messages"][0] == leo["messages"][0]     # the same instructions

# the call after a tool must start with the whole call before it
second = add_tool_result(ana, PANTRY_CALL, PANTRY_RESULT)
assert second["messages"][:2] == ana["messages"]

If someone later puts the date back on top, writes a name into the instructions or trims the tool list for one plan, the first check fails. If someone rebuilds the prompt after a tool call, the second one does.

How we did it in our own agents

In an earlier post on owning the harness we said caching deserved a post of its own. This October we made the same kind of moves on our real agents: the time went to the end of each request, the tool definitions stopped varying by project, role or language, and each tool call adds to the request instead of rebuilding it.

A tool call should add to the request, never rebuild it.

We saw even more improvement than in this small evening, because our agents often make more than one tool call for each message, and those are the calls that read the most from the cache. The more tool calls, the higher the average. Our dashboard went from the 60s to 93%. What we are building is still in development, with few people and long gaps between messages, so many calls find the cache expired, like Ana's first message of the evening; with continuous use we expect 96 to 98%.

Look at the first lines of your prompt. If they change from one call to the next, move them to the end, and let the cache do the rest.

What Google's Harness2 shows about improving an AI agent

Guillermo Baigorria — Fri, 30 Oct 2026 15:43:52 +0000

Harness² is a framework from Google Cloud AI Research and Northwestern University that improves an AI agent without touching the model. The model stays frozen; only its harness changes: the prompts, tools, skills, and scripts around it. On WorkBuddy-Bench, a benchmark of office, web, and coding tasks, that alone raised GLM-5's average score by 11.5 points.

I made the same argument in the first post of my series on building a harness: the model is rented, the harness is yours. The next parts cover how a message reaches the right agent and how to test a harness that learns from its mistakes.

Most summaries of Harness² stop at the headline. I read the paper (2026) and its code to see where the gains came from, because that is the part any team can use without running the framework.

How Harness² improves a harness without an answer key

Harness² works one task at a time. An editor model writes a few versions of the harness for the task in front of it, the same frozen model runs the task under each version, and the framework compares how the runs behaved: what each one read, did, and produced.

It then combines the changes that helped. There is no answer key: it judges by behavior.

What worked goes into a library. Each entry holds the adopted harness and a short EXPERIENCE.md file with the situation the lesson applies to, the lesson itself, and the evidence behind it. The next task starts again from the clean base harness, but the editor reads the library before proposing anything.

It does not need a stronger model to do the editing. With the same model as executor and editor, every model they tried improved, and GLM-5 editing its own harness gained 13.7 points on LAB, a benchmark of legal work.

Editing a harness without evidence made it worse

The paper also tried the shortcut: let the editor rewrite the base harness directly, without seeing any runs, any probes or the library. On coding tasks, the score fell from 73.4 to 63.2, and on web tasks from 69.9 to 65.0. Asked to improve its own setup blind, the model made it worse.

That matches what I have seen building agents. When an agent fails, the first instinct is to add a line to its prompt, before looking at what it actually received and what it did. Harness² is built the other way around: every change comes from comparing real runs.

It is careful even with its own winners. The step that writes the lessons is told that being picked is not proof an edit helped: the chosen run may have won for another reason. A lesson that rests only on mixed evidence is narrowed or dropped.

Rewriting the harness without seeing any runs dropped the coding score from 73.4 to 63.2. (Harness², 2026)

The gains were in scripts and skills, not in the prompt

In an appendix (Table 6), the authors take the improved harness and remove one group of components at a time. Removing the prompt-related components costs 1.5 points on average. Removing the skills costs 5.3, and removing the scripts 3.7.

It is one setup, Gemini 3.5 Flash on WorkBuddy-Bench, and the authors note the effects do not simply add up, but the prompt carried the least.

A script, in their terms, is code the agent runs instead of reasoning through a step token by token. A skill is a procedure for one kind of job, loaded only when that job comes up. Both are more specific than a prompt, which the model reads on every call whether it needs it or not.

In my experience, a rule added to a prompt tends to fix the case it came from, and a check in code holds every time. The paper's own case study shows it.

On a JobBench task, the agent never reopened the spreadsheets it wrote. The improved harness added, among other changes, verify_outputs.py: a script that reopens every file the agent produced and flags, for example, a formula cell a reader would see blank.

Its rule reads: confirming a file exists is not confirming it is correct. The score on that task went from 43.1 to 77.8.

Removing the skills cost 5.3 points. Removing the prompt-related components cost 1.5. (Harness², 2026)

Write the fact, not the instruction to find it

The editor's instructions are printed in the paper's appendix, and one line stands out: verified facts about the workspace are "the most valuable thing you can encode".

If the runs established that the role "HR" in one file maps to "HR Manager" in another, the editor writes that mapping into the harness, not a rule telling the agent to reconcile role names. The rule leaves the hardest step to the agent; the fact removes it.

It also labels those facts ("in this workspace...") and treats a fact copied from an earlier task as a hypothesis to check again. This is a design choice in their instructions, not a measured result, but it is the one I would copy first: many agent mistakes that look like poor reasoning are a fact the model never saw.

What Harness² leaves open

Lessons are written once. An EXPERIENCE.md entry keeps the confidence it was given when it was written, and nothing changes it when a later task reuses the lesson, and it goes badly. The authors list revising outdated experience as future work.

Its judge is the model itself. The limitations section says plainly that behavioral comparisons are "sensitive to execution variability and shared model biases", and every probe adds runs and cost.

It also answers the question of learning without retraining differently from Hermes Agent, which I read for the first post. Hermes writes skills after a session and keeps them for the next ones. Harness² starts every task from a clean harness and tests candidates before it adopts anything.

What to take from it without the framework

You can use what Harness² found without running it: look at what the agent received and did before changing anything, move what must happen the same way every time out of the prompt and into code, and give the agent the fact instead of the instruction to find it.

In every comparison in the paper, the model stayed the same. What moved the score was what the harness gave it to work with.

Rent the model, own the harness: how to build a model-agnostic AI agent

Guillermo Baigorria — Wed, 28 Oct 2026 17:57:21 +0000

Developers have Claude Code or Codex. Office workers have Claude Cowork or ChatGPT Work. Each of these is a harness: the scaffolding around an AI model that lets it do real work, with tools, context and rules. The person installing solar panels on a roof has nothing built for their work.

At Werk we built one for teams on the move, and our first decision was to keep it separate from the model: the model understands and proposes, and the harness decides what it reads, what it touches and what gets recorded. Four practices make that possible: talk to models through an interface of your own, have each agent ask for a level of work instead of a model, give every model the same format, and put what stays the same first. This post starts a series on how it is built. Why Werk exists is in Why we built Werk.

What the model does and what the harness does

Take a four-second voice note recorded on a roof: "Done with the Novak job." Getting that sentence recorded correctly takes two different kinds of work.

The model understands the request: a job is finished, and of the cards in front of it, Novak's is the one being named. The harness does everything else. It chooses which cards to show the model, checks that this person may change the card, records the change once even if the message is processed twice, and tells whoever needs to know.

A study published on arXiv in 2026 read the source code of eleven coding harnesses, including Claude Code, Codex, Gemini CLI, Hermes and Pi. It defines a harness as what connects a model to tools, context, safety, orchestration and extensions. Put simply: the model understands and proposes. The harness decides what it reads, what it touches and what gets recorded.

The same study found that none of the eleven harnesses studied imports a general-purpose agent framework; all of them run hand-written loops. Anthropic reached the same conclusion in Building effective agents (2024): the most successful implementations it saw did not use complex frameworks, and its advice is to start by calling model APIs directly.

How to keep the harness separate from the model

If the harness makes the decisions, it should not depend on any one model. Models improve every few months, no model is best at everything, and no cloud service is up 100% of the time, as providers report openly on their status pages, such as Anthropic's and OpenAI's. These are the four practices that keep the Novak note working whatever model is behind it.

We arrived at them by reading the code of more than fifteen open-source projects (MIT or Apache 2.0), such as Codex, Gemini CLI, OpenHands, Aider and Pi, and the documentation of others. We took ideas, not code.

The first is to talk to models through an interface of your own. No part of the harness calls a provider directly: the provider is configuration. We work with the major providers in the industry, and moving from one to another is a configuration change we measure before we make it.

The second is that each agent asks for a level of work, not a model. The agent that marks Novak's card as done does not know which model serves it. It declares the level it needs, and our battery of test cases decides, agent by agent, which model fills that slot.

The third is a format every model reads the same way. We follow what the industry recommends, for example the guides from Anthropic and OpenAI: each kind of data inside its own tag, so the model knows where each piece starts and ends. The conversation, the cards and the voice note reach any model in the same shape.

None of the eleven harnesses studied imports a general-purpose agent framework.

(Barbaste, Darrigol, Vu and Wiltberger, 2026)

The fourth is to put what stays the same first and what changes last. Each agent's instructions are identical on every message; the Novak note is not. With the stable part first, the provider reuses what it has already processed, so the answer arrives sooner and costs less: with DeepSeek Harness, one user measured a 98% cache hit rate across 650 requests. Cache optimization deserves a post of its own, and we will write it.

What no SDK can solve for you

Agent SDKs such as the OpenAI Agents SDK and the Claude Agent SDK are a great way to start: they handle the cycle of calling the model, running tools and repeating. We studied them closely, along with LangGraph, the Vercel AI SDK, Pydantic AI and Mastra. But the hardest part of the Novak note is not in that cycle.

It is in what belongs to a team. Novak's card lives in a record that several people share, each with their own permissions. If Werk asks which of two cards was meant and the answer comes back as "the second one," that answer has to stay tied to what that person saw on screen. That depends on each product, and the good SDKs say so plainly: Mastra's documentation recommends binding an approval to the exact arguments the approver saw.

A personal agent is not enough either. Hermes Agent and OpenClaw, two of the open-source projects we read, are built for one person. Hermes says so in its security policy: it is a single-tenant personal agent, and everyone with access is equally trusted. For a personal agent, that is the right call. On a team, each person sees their own work and what the project shares, and the model never decides who sees what.

The model understands and proposes. The harness decides what it reads, what it touches and what gets recorded.

That is why we built our own harness, and why the next part of this series covers its central piece: the router, which receives the Novak note and hands it to the right agent.

The model changes, the harness stays

Because the important decisions live in the harness, the model can be chosen, compared and replaced when a better one comes along. The harness is what is yours: what each agent reads, what the code checks before touching the record, how an error is caught and how it is fixed without breaking something else.

Models will keep getting better. With a well-separated harness, each improvement reaches daily use on its own: a voice note understood better, the right card, the notice to whoever needs to know.

Teams that use Werk do not have to do anything to benefit. The harness does that work.

The ideas and decisions are mine; I used AI to help edit the text.