DEV Community: Anis Meziani

How to cut your AI agent's input tokens: what worked, what failed, and a bench to measure yours

Anis Meziani — Wed, 28 Oct 2026 14:40:29 +0000

Before every single thought, our BI agent re-read 17,000 tokens of instructions and tool manuals.

Then the whole conversation. Then every table it had pulled since the first question. In our offline replay of a three-question conversation, ten model calls consumed 286,522 input tokens before optimization. Real runs behaved differently, as you'll see below.

Nobody had decided that. It's just what an agent does by default: every model call starts from zero, so everything the agent "knows" gets sent again. Every call, every turn, every retry.

This post is a field report on how we brought it down: how we measured, which cuts paid off and in what order, what didn't work, and a checklist you can run on your own agent tomorrow. It ends with five questions we can't answer alone. The optimizations reported here are implemented, and the benchmark is open source, so you can check every number. Some experiments and benchmark improvements are still in progress, and I'll say which.

It's the third post in a series about what breaks when AI agents run in production, after agents that forgot everything after a restart and agents that redid work they had already done. This one is about the opposite of amnesia: agents that remember too much, and pay for it on every call.

TL;DR: six lessons

Measure before you cut. Without a bench and quality checks, our best-looking cut would have quietly broken the answers.
Start with the tools. Instructions plus 36 tool declarations cost 17,000 tokens before the question. Removing the tools the agent never calls was the cheapest win.
Forget the past, never the present. Shortening old tool results saved 21 to 26%. Shortening the current turn broke quality on 4 scenarios.
Stable prompts are cacheable prompts. In our benchmark, ordering the schema tables deterministically raised the reusable prefix of a SQL prompt from 11% to 98%, with the same tokens.
The biggest real gain came from fewer calls, not smaller ones: −41% on a question, because the model asked once instead of five times.
Your bench will lie to you a little. Ours overestimated real gains by 9 to 16%, underestimated one by a lot, and missed a failure entirely.

1. Measure before you cut

Know what you're paying for

Here's what our BI agent sent to the model on every call, before we changed anything:

Layer	What it is	Paid when
Instructions	the agent's system prompt	every call
Tool declarations	name, description and schema of each of 36 tools	every call
History	every message of the conversation	every call
Past tool results	every table, file and query result pulled so far	every call
Current turn	the question, and the results the model is working on now	every call of this turn

The first two layers alone came to about 17,000 tokens: the whole first call weighed 17,070, a one-line question included. The model paid them before reading a word of the question, and again at each of the 3 to 5 calls a question takes.

We're not alone. The industry now calls it the "MCP tax": one measurement published in May found around 71,000 tokens of overhead for seven MCP servers, about 35% of a 200K context window, before the first prompt. And the cost isn't only money: research on context rot keeps showing that models get worse as their input grows, well before the window is full. A useless token is paid twice, on the bill and in the answer.

Build the bench first

We built a bench before touching anything. It lives in bench/bi_tokens:

Five scenarios of our BI agent: creating a dashboard, asking a question about it, chatting with it, a three-question conversation, and a forecast.
An offline replay with a scripted model, so each change is compared on identical calls, input tokens counted call by call.
Quality checks on every scenario: the right facts, grounded in the data. A cut that saves tokens but fails a check is rejected.
Real runs afterwards, with costs read from the usage table where every model call is recorded.

That last part matters more than it sounds. Section 3 explains why.

2. The cuts, ranked by effort and gain

Here's what we did, in the order we'd do it again:

Cut	Effort	Measured gain	Watch out for
Remove the tools the agent never calls	low	−10 to −15% per scenario, first call −18.5% on real runs	nothing observed
Shorten old tool results, past turns only	low	−21 to −26% on conversations	never touch the current turn
Describe big payloads instead of pasting them	low	base64 page images no longer re-sent at every turn	the agent must still reach the full data through tools
Make the prompt prefix cache-friendly	low	reusable prefix 11% → 98% (benchmark, upper bound)	depends on your provider and on which tables are selected
Fewer round trips	medium	−41% on a question, real run	only pays if the model uses it
Trim the instructions	low	about −2%	small, do it last

Remove the tools the agent never calls

Our BI templates had inherited the generic chat tools of the platform, the ones a general assistant needs and a BI agent never calls. Most agent stacks ship this way; we just measured ours.

We removed ten of them from the BI templates, and trimmed what remained: declarations lost their pydantic titles and "Returns" sections, and tool responses are capped. 36 tools down to 26, and the first call went from 17,070 to 13,598 tokens on the bench. On real runs, together with the instruction trim described below, it went from about 14,500 to 11,800.

Lesson: an agent's toolbox is a cost line. Declare tools per agent, not per platform.

Shorten old tool results, never the current turn

After a few questions, most of the context was old tool results: tables the agent had pulled two turns ago and already answered from. Now, in every turn before the current one, a large tool result keeps its small fields (ids, titles, status, a message) and loses its bulk, with a note telling the model it can call the tool again if it needs the data.

def compact_previous_turns(contents):
    # The current turn starts at the last user message that isn't a tool result.
    current = current_turn_start(contents)
    for content in contents[:current]:          # earlier turns only
        for part in content.parts:
            if is_large_tool_result(part):
                part.response = keep_small_fields(part.response)  # ids, titles, status

It runs on a copy: the stored session keeps everything, only what the model re-reads is shortened. And the answer the model gave at the time stays in the history, so what it concluded isn't lost.

Lesson: storing everything is cheap. Re-sending everything is what costs.

Describe big payloads instead of pasting them

Big files. A large CSV used to be pasted into the context. Now the agent gets the header, the first 20 rows and the row count, and works on the full data through its tools.
Images. In our pipeline, a base64-encoded PDF page was estimated at roughly 100,000 to 150,000 text tokens, depending on the page and its encoding (an estimate written down in our code when we hit it, not a formal benchmark), and it used to be re-sent at every later turn. Now images go to the model as images, and base64 is stripped from older turns. The model saw the page once; its own description of it is the source of truth after that.

Lesson: the model needs to know what's in a file, not to hold the file.

Make the prompt prefix cache-friendly

Providers cache prompt prefixes, but only if the beginning of your request is exactly the same as before. Our text-to-SQL tool injected the relevant tables of the schema in order of relevance. Relevance changes with every question, so the prefix did too.

We now write the selected tables in name order: same tables, same tokens, only the order changes. Share of each request that repeats the previous request's prefix, in our benchmark:

Schema	Relevance order	Name order
25 tables, all selected every time	11%	98%
80 tables, 62 or 63 selected per question	11%	93%

How we measured this, and what the number is not

The method. The benchmark sends 10 fixed questions against a fixed schema. For questions 2 to 10, it counts how much of each request matches the previous request's prefix, using OpenAI's caching rules: prefixes of at least 1,024 tokens, counted in 128-token blocks. The tokens sent and the tables selected are identical before and after; only the order changes. The script is bench/sql_prompt_cache/measure.py.

What the number is not.

Not the share actually served from the cache. That depends on the provider routing your requests to the same cache, and only shows up in the cached tokens your provider reports. Ours is an upper bound.
Not a cost reduction. Cached tokens are cheaper, not free, and pricing differs by provider.
Not a guarantee. Ordering alone doesn't make a prefix stable. It works best when the same tables come back from one question to the next, which is why the 80-table row is lower.

Two more things we learned the hard way:

Some providers cache nothing unless you ask. On Claude models, nothing was cached until we set cache markers: every call of a turn paid the instructions and tool declarations, 5,000 to 17,000 tokens depending on the agent, at the full rate.
Count cached tokens for what they cost. Our usage quotas counted cached input at full price. They now count it at 25%, otherwise users are charged for savings already made.

Lesson: keep the stable parts of your prompt stable. When your provider supports prompt caching, deterministic ordering can turn repeated context into reusable context.

Fewer round trips

Every extra model call re-pays the whole stack. So the cheapest call is the one that doesn't happen.

One grouped query instead of five. We gave the dashboard query tool a group_by parameter. On a real run, the model asked one grouped question instead of one query plus four filtered ones: from 71,617 to 42,130 input tokens, −41.2%, same correct answer.
Repair inside the tool, not in the agent's loop. When the database rejects a generated SQL query, the SQL tool now regenerates it once with the database error, instead of sending the error back to an agent that would re-read its whole context to try again.
Query and chart in one call for our Data Analyst agent, instead of two.

The catch: this kind of cut only pays if the model actually uses it. group_by adds 72 tokens to every call's declarations whether or not the model reaches for it. On real runs, it reached for it on 4 of 7 dashboard queries.

Lesson: look at how many calls a question takes before you look at how big each call is.

Trim the instructions, last

Removing an instruction block that described tools the agent didn't have saved about 2%. Worth doing, because the model was being told about tools it couldn't call. But it's the smallest win on this list, so it comes last.

3. What didn't work, or not the way we expected

This is the part I'd want to read in someone else's post.

Compacting the current turn. It looked like the obvious next step after shortening old results. The quality checks on facts and grounding failed on 4 scenarios. Within a turn, the model is still working on its results; take them away and it starts answering from memory. Without the bench, we'd have shipped it.

Trusting the bench too much. Our scripted replay overestimated real totals by 9 to 16%, because a real model doesn't always take the same path as a script. It also underestimated one gain by a lot: −13.8% on the bench for a dashboard question, −41.2% for real, because the real model changed its behavior (one grouped query) in a way a script can't.

A blind spot we didn't see. On real runs, the forecast scenario failed: the orders file has several rows per date, and our forecast engine expects one and doesn't aggregate them. None of the cuts caused it, but the bench never saw it, because it replays a canned forecast answer instead of calling the real engine. We left the forecast numbers out of the results, and aggregating by date before forecasting is on our list.

A cache we don't control. Depending on the session, between 17% and 66% of our input was already served from the provider's implicit cache, without us doing anything. Nice when it happens, impossible to plan for. That's why stable ordering matters: it turns luck into something you can count on.

The fix we almost made. Twice in one real run, the model tried to group by a period filter the tool didn't accept for grouping. Those two failed attempts cost about 27,000 input tokens. The reflex was a stricter docstring. We did the opposite: the tool now accepts what the model reached for, and its error lists the values it does accept. When a model keeps reaching for something, that's often a spec for the tool, not a mistake to forbid.

4. Memory that earns its tokens

None of this means agents should remember less. It means every token they re-read has to earn its place.

Sessions live in the database, so the conversation survives restarts. That was the point of the first post.
A window without amnesia. Our data and BI agents send a window of recent messages, and the text of the turns that fall out of it is kept: the agent forgets the bulk of what it pulled, not what was said.
Long-term memory, scoped per agent and per user, so one user's memories never leak into another's.
A case bank, on trial. Inspired by the Memento paper, each finished run becomes a case (task, tools called, outcome), and similar cases are shown before the next run. It's behind a flag, because memory isn't free either. We wrote the go/no-go down before the first real measurement: at least +10 points of success rate with 4 past cases, no rise in tool errors, at most +10% input tokens per task. The results aren't in yet.

And one guardrail ties it all together: a retry re-pays everything. When a provider hits its quota, our whole queue pauses instead of retrying every few seconds at full input cost, and a run is capped at 25 model calls, so a loop can't spin forever on your bill.

5. The results

Offline bench, all cuts combined (input tokens, scripted model, quality checks passing on these scenarios):

Scenario	Before	After	Change
Dashboard creation	104,285	86,925	−16.6%
Dashboard question	75,227	64,811	−13.8%
Dashboard chat	151,177	98,321	−35.0%
Three-question conversation	286,522	196,983	−31.3%
First call	17,070	13,598	−20.3%

Real runs (input tokens read from our usage table, correct answers):

Measure	Before	After	Change
First call floor	~14,500	~11,800	−18.5%
Dashboard creation	82,212	63,526	−22.7%
Dashboard question	71,617	42,130	−41.2%

6. A checklist for your own agent

If you only take one thing from this post, run these seven checks:

Weigh your first call. How many tokens does your agent send before the user's question? If you don't know, start there.
Compare declared tools with called tools. Over your last few hundred runs, which tools were never called? Remove them from that agent.
Look at what you re-send from past turns. Shorten old tool results. Never the current turn.
Find the files and images in your context. Send a preview and let the agent reach the rest through tools. Send images as images.
Check that your prefix is stable. Order what you inject deterministically, set cache markers where your provider needs them, and check the cached tokens your provider actually reports.
Count calls per question. Look for loops and retries that re-pay the whole stack, and cap calls per run.
Build a bench with quality checks before you cut, and confirm every gain on real runs.

Reproduce it, and make the bench yours

1. Reproduce our measurements. The bench, the scenarios and the results are in the repo:

git clone https://github.com/apowerb/apowerb && cd apowerb
python bench/bi_tokens/replay.py --estimates --compare bench/bi_tokens/results/main.json

2. If you want to go further, contribute. The bench only knows our agents today. You can add a scenario that looks like your traffic (one JSON file in bench/bi_tokens/scenarios), make the forecast scenario call the real engine so the bench can't miss a failure like ours again, or fix the forecast by aggregating repeated dates. Comment on GitHub before you start, so we agree on the shape. The tech team reviews every PR, and contributors are credited by name.

Source code: github.com/apowerb/apowerb
Run it locally (Docker Compose) or on Kubernetes (Helm): github.com/apowerb/apowerb-hosting
Documentation: docs.apowerb.com
Try it without installing anything: apowerb.com

The questions we don't have good answers to yet

This is where I'd genuinely like your experience. The last two posts got better because readers answered questions like these.

Static tools or tools on demand? We declare a fixed toolbox per agent. The other school loads tool definitions only when the model asks for them. That saves the declarations, but costs an extra call to find the right tool. Where does on-demand start to pay: 30 tools, 50, 100? Have you measured it?
Where does compaction stop? We never touch the current turn. But some turns pile up ten tool results before the model answers. Do you summarize inside a long turn, with a model that costs tokens itself, or truncate with a rule that might drop the one row that mattered?
Implicit or explicit cache? Between 17% and 66% of our input came from our provider's implicit cache, depending on the session. We have an explicit cache wired in but turned off by default, because it means managing lifetimes and paying to write the cache. Has an explicit cache ever paid for itself in your agents, and at what traffic?
How much should a cached token count? We weight cached input at 25% in our quotas. But providers price it differently, and some charge a premium to write the cache, which we don't track yet. How do you bill or budget cached tokens for your users?
How do you build a bench that can't lie? Ours replays canned tool answers to stay cheap and repeatable, and that's exactly how it missed a real failure. Do you call real engines in your bench, and how do you keep it affordable and stable?

Answer in the comments, even with a single number. The best answers will shape the next changes in the repo, credited to you.

And if you only have time for one: how many tokens does your agent pay before it reads the user's question?

Before every single thought, our BI agent re-read 17,000 tokens of instructions and tool manuals.

Nobody had decided that. It's just what an agent does by default: every model call starts from zero, so everything the agent "knows" gets sent again. Every call, every turn, every retry.

TL;DR: six lessons

Measure before you cut. Without a bench and quality checks, our best-looking cut would have quietly broken the answers.
Start with the tools. Instructions plus 36 tool declarations cost 17,000 tokens before the question. Removing the tools the agent never calls was the cheapest win.
Forget the past, never the present. Shortening old tool results saved 21 to 26%. Shortening the current turn broke quality on 4 scenarios.
Stable prompts are cacheable prompts. In our benchmark, ordering the schema tables deterministically raised the reusable prefix of a SQL prompt from 11% to 98%, with the same tokens.
The biggest real gain came from fewer calls, not smaller ones: −41% on a question, because the model asked once instead of five times.
Your bench will lie to you a little. Ours overestimated real gains by 9 to 16%, underestimated one by a lot, and missed a failure entirely.

1. Measure before you cut

Know what you're paying for

Here's what our BI agent sent to the model on every call, before we changed anything:

Layer	What it is	Paid when
Instructions	the agent's system prompt	every call
Tool declarations	name, description and schema of each of 36 tools	every call
History	every message of the conversation	every call
Past tool results	every table, file and query result pulled so far	every call
Current turn	the question, and the results the model is working on now	every call of this turn

Build the bench first

We built a bench before touching anything. It lives in bench/bi_tokens:

Five scenarios of our BI agent: creating a dashboard, asking a question about it, chatting with it, a three-question conversation, and a forecast.
An offline replay with a scripted model, so each change is compared on identical calls, input tokens counted call by call.
Quality checks on every scenario: the right facts, grounded in the data. A cut that saves tokens but fails a check is rejected.
Real runs afterwards, with costs read from the usage table where every model call is recorded.

That last part matters more than it sounds. Section 3 explains why.

2. The cuts, ranked by effort and gain

Here's what we did, in the order we'd do it again:

Cut	Effort	Measured gain	Watch out for
Remove the tools the agent never calls	low	−10 to −15% per scenario, first call −18.5% on real runs	nothing observed
Shorten old tool results, past turns only	low	−21 to −26% on conversations	never touch the current turn
Describe big payloads instead of pasting them	low	base64 page images no longer re-sent at every turn	the agent must still reach the full data through tools
Make the prompt prefix cache-friendly	low	reusable prefix 11% → 98% (benchmark, upper bound)	depends on your provider and on which tables are selected
Fewer round trips	medium	−41% on a question, real run	only pays if the model uses it
Trim the instructions	low	about −2%	small, do it last

Remove the tools the agent never calls

Our BI templates had inherited the generic chat tools of the platform, the ones a general assistant needs and a BI agent never calls. Most agent stacks ship this way; we just measured ours.

Lesson: an agent's toolbox is a cost line. Declare tools per agent, not per platform.

Shorten old tool results, never the current turn

def compact_previous_turns(contents):
    # The current turn starts at the last user message that isn't a tool result.
    current = current_turn_start(contents)
    for content in contents[:current]:          # earlier turns only
        for part in content.parts:
            if is_large_tool_result(part):
                part.response = keep_small_fields(part.response)  # ids, titles, status

It runs on a copy: the stored session keeps everything, only what the model re-reads is shortened. And the answer the model gave at the time stays in the history, so what it concluded isn't lost.

Lesson: storing everything is cheap. Re-sending everything is what costs.

Describe big payloads instead of pasting them

Big files. A large CSV used to be pasted into the context. Now the agent gets the header, the first 20 rows and the row count, and works on the full data through its tools.
Images. In our pipeline, a base64-encoded PDF page was estimated at roughly 100,000 to 150,000 text tokens, depending on the page and its encoding (an estimate written down in our code when we hit it, not a formal benchmark), and it used to be re-sent at every later turn. Now images go to the model as images, and base64 is stripped from older turns. The model saw the page once; its own description of it is the source of truth after that.

Lesson: the model needs to know what's in a file, not to hold the file.

Make the prompt prefix cache-friendly

We now write the selected tables in name order: same tables, same tokens, only the order changes. To measure the effect, our benchmark sends 10 fixed questions against a fixed schema, and counts, for questions 2 to 10, how much of each request matches the previous request's prefix, using OpenAI's caching rules (prefixes of at least 1,024 tokens, in 128-token blocks):

Schema	Reusable prefix, relevance order	Reusable prefix, name order
25 tables, all selected every time	11%	98%
80 tables, 62 or 63 selected per question	11%	93%

Three things this number is not. It's not the share actually served from the cache: that depends on the provider routing your requests to the same cache, and shows up in your usage data. It's not a cost reduction either. And ordering alone doesn't guarantee a stable prefix: it works best when the same tables come back from one question to the next, which is why the second row is lower.

Two more things we learned the hard way:

Some providers cache nothing unless you ask. On Claude models, nothing was cached until we set cache markers: every call of a turn paid the instructions and tool declarations, 5,000 to 17,000 tokens depending on the agent, at the full rate.
Count cached tokens for what they cost. Our usage quotas counted cached input at full price. They now count it at 25%, otherwise users are charged for savings already made.

Lesson: keep the stable parts of your prompt stable. When your provider supports prompt caching, deterministic ordering can turn repeated context into reusable context.

Fewer round trips

Every extra model call re-pays the whole stack. So the cheapest call is the one that doesn't happen.

One grouped query instead of five. We gave the dashboard query tool a group_by parameter. On a real run, the model asked one grouped question instead of one query plus four filtered ones: from 71,617 to 42,130 input tokens, −41.2%, same correct answer.
Repair inside the tool, not in the agent's loop. When the database rejects a generated SQL query, the SQL tool now regenerates it once with the database error, instead of sending the error back to an agent that would re-read its whole context to try again.
Query and chart in one call for our Data Analyst agent, instead of two.

Lesson: look at how many calls a question takes before you look at how big each call is.

Trim the instructions, last

3. What didn't work, or not the way we expected

This is the part I'd want to read in someone else's post.

4. Memory that earns its tokens

None of this means agents should remember less. It means every token they re-read has to earn its place.

Sessions live in the database, so the conversation survives restarts. That was the point of the first post.
A window without amnesia. Our data and BI agents send a window of recent messages, and the text of the turns that fall out of it is kept: the agent forgets the bulk of what it pulled, not what was said.
Long-term memory, scoped per agent and per user, so one user's memories never leak into another's.
A case bank, on trial. Inspired by the Memento paper, each finished run becomes a case (task, tools called, outcome), and similar cases are shown before the next run. It's behind a flag, because memory isn't free either. We wrote the go/no-go down before the first real measurement: at least +10 points of success rate with 4 past cases, no rise in tool errors, at most +10% input tokens per task. The results aren't in yet.

5. The results

Offline bench, all cuts combined (input tokens, scripted model, quality checks passing on these scenarios):

Scenario	Before	After	Change
Dashboard creation	104,285	86,925	−16.6%
Dashboard question	75,227	64,811	−13.8%
Dashboard chat	151,177	98,321	−35.0%
Three-question conversation	286,522	196,983	−31.3%
First call	17,070	13,598	−20.3%

Real runs (input tokens read from our usage table, correct answers):

Measure	Before	After	Change
First call floor	~14,500	~11,800	−18.5%
Dashboard creation	82,212	63,526	−22.7%
Dashboard question	71,617	42,130	−41.2%

6. A checklist for your own agent

If you only take one thing from this post, run these seven checks:

Weigh your first call. How many tokens does your agent send before the user's question? If you don't know, start there.
Compare declared tools with called tools. Over your last few hundred runs, which tools were never called? Remove them from that agent.
Look at what you re-send from past turns. Shorten old tool results. Never the current turn.
Find the files and images in your context. Send a preview and let the agent reach the rest through tools. Send images as images.
Check that your prefix is stable. Order what you inject deterministically, set cache markers where your provider needs them, and check the cached tokens your provider actually reports.
Count calls per question. Look for loops and retries that re-pay the whole stack, and cap calls per run.
Build a bench with quality checks before you cut, and confirm every gain on real runs.

The questions we don't have good answers to yet

This is where I'd genuinely like your experience. The last two posts got better because readers answered questions like these.

Static tools or tools on demand? We declare a fixed toolbox per agent. The other school loads tool definitions only when the model asks for them. That saves the declarations, but costs an extra call to find the right tool. Where does on-demand start to pay: 30 tools, 50, 100? Have you measured it?
Where does compaction stop? We never touch the current turn. But some turns pile up ten tool results before the model answers. Do you summarize inside a long turn, with a model that costs tokens itself, or truncate with a rule that might drop the one row that mattered?
Implicit or explicit cache? Between 17% and 66% of our input came from our provider's implicit cache, depending on the session. We have an explicit cache wired in but turned off by default, because it means managing lifetimes and paying to write the cache. Has an explicit cache ever paid for itself in your agents, and at what traffic?
How much should a cached token count? We weight cached input at 25% in our quotas. But providers price it differently, and some charge a premium to write the cache, which we don't track yet. How do you bill or budget cached tokens for your users?
How do you build a bench that can't lie? Ours replays canned tool answers to stay cheap and repeatable, and that's exactly how it missed a real failure. Do you call real engines in your bench, and how do you keep it affordable and stable?

Answer in the comments, even with a single number. The best answers will shape the next changes in the repo, credited to you.

Reproduce it, then tell us your numbers

1. Reproduce our measurements. The bench, the scenarios and the results are in the repo:

git clone https://github.com/apowerb/apowerb && cd apowerb
python bench/bi_tokens/replay.py --estimates --compare bench/bi_tokens/results/main.json

2. Share your own numbers in the comments. Run the checklist on your agent and post what you find, even just the first one: how many tokens it pays before the user's question. The last two posts got better because readers answered, and I'd bet this one will too.

3. If you want to go further, contribute. The bench only knows our agents today. You can add a scenario that looks like your traffic (one JSON file in bench/bi_tokens/scenarios), make the forecast scenario call the real engine so the bench can't miss a failure like ours again, or fix the forecast by aggregating repeated dates. Comment on GitHub before you start, so we agree on the shape. The tech team reviews every PR, and contributors are credited by name.

How many tokens does your agent pay before it reads the user's question?

Code : https://github.com/apowerb/apowerb
Docs : https://docs.apowerb.com/
Deploy with Docker Compose or Helm : https://github.com/apowerb/apowerb-hosting
Try it out : https://apowerb.com

Idempotency keys for AI agents: why hashing the arguments doesn't work

Anis Meziani — Wed, 21 Oct 2026 15:00:45 +0000

After our last post on why our agents refuse to redo work after a crash, a reader left a comment that took the discussion a level deeper.

The point, roughly: you can't cleanly resume a non-deterministic trajectory, so the only thing that really holds is idempotency at the effect boundary. And then the question that stuck with me: why make the orchestrator decide what's safe to replay, instead of pushing that decision down to the tools themselves?

They're right, and this post is my attempt to answer properly. It covers what we do today, why the obvious version of that idea breaks with LLMs, and the design we're converging on. Fair warning: the last part isn't shipped. It's a proposal, and I'd genuinely like it torn apart.

TL;DR

Today, ApowerB refuses to auto-replay any run that called any tool, reads included. Safe, blunt, and stuck the moment a run completes halfway.
The fix is per-tool idempotency keys, so a replay dedupes at the tool. But hashing the arguments doesn't work when an LLM writes them: the replay drafts a slightly different email, and the hash changes.
The rule that holds: build the key from stable intent, never from generated content.
We already do this in two places, and enforce "read-only" by construction in a few others. The rest is a design sketch, and an open invitation.

Where we are today: default-deny

When a run fails, our runtime records which tools it called. If it called anything, an automatic replay is refused with a 409 that lists the tools (run_main.py). A human can still force it.

We don't classify tools. A GET counts the same as a send_email. That sounds lazy, so let me defend it for a second, because the reader's own examples are the best argument for it:

a GET against a rate-limited or stateful API is not free to repeat;
a "pure" lookup that writes an audit row isn't pure;
a tool from a third-party MCP server can call itself read-only, and you have no way to check.

Inferring purity gets exactly these cases wrong, and they're the ones that hurt. Refusing everything costs a click. Guessing wrong costs a duplicate email to a customer.

Where default-deny breaks

Default-deny is safe. It is also stuck the moment a run finishes halfway.

Picture an email campaign. Our email-campaign skill already puts a human gate in front of the effect: it drafts everything, then never sends without explicit approval. Once approved, it loops over the recipients and sends one email each.

Now the pod dies after email 40 of 100.

Replay is refused, because tools were called. Correct, and useless: 60 people never get their email.
Force the replay, and 40 people get it twice.

There is no good answer at the run level. The run is the wrong unit. The email is the unit. That's exactly where the reader's idea comes in: let each effect know whether it already happened.

The obvious version, and why it breaks

The textbook approach: every effectful tool computes an idempotency key from its arguments, say sha256(json(args)), and the runtime skips any call whose key it has already executed.

With a normal program, that works. With an LLM it quietly fails, because the arguments aren't stable either.

First attempt, the model calls:

{"to": "anna@acme.com", "subject": "Following up on your request",
 "body": "Hi Anna, following up on the quote you asked for..."}

Replay, same input, same agent:

{"to": "anna@acme.com", "subject": "Your quote",
 "body": "Hello Anna, I'm getting back to you about your quote..."}

Different subject, different body, different hash. The ledger has never seen this key. Anna gets two emails, and your idempotency layer reports that everything went perfectly.

The reader's line from the last thread applies one level down: step 3 isn't a stable coordinate. Neither are the arguments.

The rule: stable intent, never generated content

Split every argument into three kinds:

Kind	Examples	In the key?
Identity from the outside world	event id, message id, thread id, record id, normalized recipient address	Yes
Derived deterministically from inputs	hash of an uploaded file's content	Yes
Generated by the model	body, subject, summary, any free text	Never

The key answers "have I already done this thing, for this reason, to this target?", not "have I already sent these exact bytes?". For the campaign: key = (triggering_event, tool, recipient). The replay can rewrite every email from scratch; Anna's key is the same, so she's skipped.

Two places where we already do this

This isn't hypothetical in our codebase. It's just not generalized yet.

1. Webhooks: the key is the provider's identity. When Microsoft Graph re-delivers a notification, a unique index on (subscription_id, resource_id) rejects the duplicate insert, and the message is processed once (outlook.py). Nothing generated goes into that key. It's pure outside-world identity.

2. BI datasets: the key is derived from the content. When an agent imports a file as a dataset, the dataset's ID isn't random. It's a UUIDv5 over who owns it, where it lives, and a SHA-256 of the bytes (bi_datasets.py):

digest = hashlib.sha256(attachment["content"]).hexdigest()
dataset_id = str(uuid.uuid5(
    uuid.NAMESPACE_URL,
    "|".join([owner.lower(), organization_id, project_id,
              attachment["folder"], attachment["name"],
              data.sheet or "", digest]),
))
existing = await store.get(dataset_id)
if existing is not None:
    return {"success": True, "dataset_id": dataset_id, "reused": True}

Run it twice on the same file: one dataset. Change one byte: a new one. Notice what's not in the key: the display name, which is formatted text.

And here's the scar that comes free with it. What if the user deleted that dataset? Soft deletes keep the row, so the deterministic ID is still taken, and the insert collides. Our code catches the IntegrityError and falls back to a fresh random ID. That's the general lesson: an idempotency key outlives the thing it points to. Your ledger needs to know the difference between "already done" and "done, then undone".

And some tools are pure by construction, not by declaration

The other half of the reader's question was classification. Our answer so far: we don't trust a tool that says it's read-only. We trust one that can't write.

The Oracle tool opens a SET TRANSACTION READ ONLY and rolls back (oracle.py).
The MongoDB tool refuses aggregation pipelines with a write stage ($out, $merge) (mongodb.py).
The Redshift tool validates that the SQL is read-only before it runs (redshift.py).

These are the only tools we'd be comfortable marking "safe to replay", and only because the guarantee lives in the tool's code, not in a flag next to it.

The design we're converging on

This part is a sketch, not shipped code. It's the shape we think is right, written down so it can be challenged.

1. Tools declare their effect, and default to the worst case.

@tool(effect="none")            # pure by construction, enforced in the tool
def oracle_run_query(sql: str): ...

@tool(effect="idempotent",
      key=lambda args, ctx: (ctx.trigger_event_id, "send_email",
                             normalize(args["to"])))
def send_email(to: str, subject: str, body: str): ...

@tool                            # no declaration: treated as unsafe
def third_party_mcp_action(**kwargs): ...

2. An effects ledger, with a unique constraint. Before an idempotent tool runs, the runtime looks its key up in a ledger table. Already there: return the stored result, don't execute. Not there: execute, then record. The same unique-index trick as our webhooks, applied to every effect.

3. Undeclared stays default-deny. Anything without a declaration, including every third-party tool, keeps today's behavior: if it ran, the run isn't auto-replayed. Declaring a tool idempotent is how you earn automatic replay, one tool at a time.

4. Pass the key through when the provider supports it. Some APIs accept an idempotency key natively (Stripe's Idempotency-Key header is the classic example). When they do, the key should travel all the way to the provider, so even a crash between the API call and our ledger write dedupes on their side.

With that in place, the campaign that died at email 40 replays cleanly: 40 ledger hits, 60 sends, nobody gets two emails.

The questions I don't have good answers to yet

This is where I'd genuinely like help:

Which fields go in the key, per tool? For email, (event, recipient) feels right. For "post a summary to Slack", is it (event, channel)? What about an agent that legitimately sends two different emails to the same person for the same event?
What if the replay takes a different branch? The model may decide to email a different set of people the second time. Per-target keys handle overlap, but what about the people who were emailed in run one and dropped in run two?
How long does the ledger remember? Forever is wrong (storage, and "the same email next month" is legitimate). Too short reopens the duplicate window.
The crash between the effect and the ledger write. If the provider has no native idempotency, there's a gap. Is "record intent before, confirm after" plus reconciliation good enough?

Come build it with us

If you've shipped something like this, or you have strong opinions about any of the questions above, we'd love your help. We're opening an RFC issue on tool-level idempotency keys, and every contributor is credited by name.

Source code: github.com/apowerb/apowerb
Issues and RFCs: github.com/apowerb/apowerb/issues
Run it locally (Docker Compose) or on Kubernetes (Helm chart): github.com/apowerb/apowerb-hosting
Helm chart on Artifact Hub: artifacthub.io/packages/helm/apowerb/apowerb-chart
Documentation: docs.apowerb.com
Product page: thaink2.com/apowerb

And thanks to the reader whose comment started all of this. The best part of writing in public is when the comments take the topic somewhere new.

How do you build idempotency keys for your agent's tools? Hash the args, stable fields, something else? Tell me in the comments.

[Boost]

Anis Meziani — Wed, 14 Oct 2026 09:53:50 +0000

David Elom GNAGLO

Oct 14

Running ADK agents with MCP tools from a REST API

#python #ai #agents #opensource

3 min read

Amnesia and déjà vu: how our AI agents failed after every restart (and how we fixed it)

Anis Meziani — Tue, 13 Oct 2026 15:14:02 +0000

An agent that crashes is annoying.

An agent that crashes and comes back is where things get dangerous.

When we started running our agents in production, we discovered that a restart could go wrong in two completely opposite ways. Sometimes the agent came back with amnesia: it had forgotten everything, the conversation, the context, what it was in the middle of doing. And sometimes, when we tried to fix that by simply relaunching the work, it came back with déjà vu: it happily redid everything from the start, including the things it had already done in the real world.

Like sending an email. Twice.

This post is about both failures, why the obvious fixes don't work, and what we ended up building. It's the part of an agent nobody puts in a demo, and it's the part that decides whether you can trust it.

Everything below comes from ApowerB, the open-source agent runtime we build at thaink², and every fix links to the real code. If you're new to the project, this overview of the architecture is a good place to start.

TL;DR

Amnesia: by default, our agent framework kept every conversation in RAM. Every restart erased everything. We moved the agent's memory into Postgres.
Déjà vu: "just relaunch it" is a trap. An agent acts on the world through tools. Relaunching blindly repeats those actions.
You can't cleanly "resume from step 3" with an LLM agent. So we stopped asking how do we resume? and started asking is it safe to start again?
Every run is now written down when it starts, with its input. Every tool it calls is recorded. A failed run can be replayed without a second thought only if we know it touched nothing. Otherwise the replay is refused, loudly, with the list of what it already did, until a human decides.

Part 1: Amnesia

Eight restarts a day, zero memory

At some point we ran an internal audit on one of our deployments, and one number jumped out: around eight restarts a day.

That's not unusual. Every deploy restarts the app. Kubernetes moves pods around. A node gets drained. In a healthy cluster, processes come and go all the time.

What was a problem: after each of those restarts, every conversation was gone. No history, no context. A user could be halfway through a task with an agent, and the next message would land on an agent that had never heard of them.

The cause was a default. We build on Google's Agent Development Kit (ADK). Each agent conversation lives in a session: the list of events that happened (what the user said, what the model answered, which tools it called, what came back). That session is the agent's short-term memory. And if you don't configure where sessions live, ADK falls back to an in-memory session service. A Python dict, basically. It works perfectly on your laptop, and it evaporates every time the process restarts.

The tab that killed the workflow

The second kind of amnesia was even sillier.

Our visual workflow builder (you chain agents on a canvas, and each one passes its output to the next) originally ran in the browser. The browser was the orchestrator. Which meant that if you closed the tab in the middle of a long workflow, the workflow just... stopped. Halfway. With no record of where it was.

Runs that left nothing behind

And the third one was the most fundamental.

Only one path in the whole product could retry a failed run: incoming emails, because the webhook queue kept the payload and an attempt counter. Everything else (a workflow launched from the canvas, a scheduled agent, an API call) left nothing behind when it failed. The workflow router literally kept its state in a Python dict that was emptied at the end of the stream.

So when something failed, we couldn't relaunch it, because we didn't know what it had been asked to do. We had traces, logs, token counts. We could see the failure in detail. We just couldn't redo the work.

That was the moment it clicked for us:

What was missing wasn't observability. It was the input.

Fix 1: give the agent a memory that survives restarts

The first fix is one setting, and it's the most important line in this post. We point ADK's session service at Postgres instead of RAM (see it in main.py):

app = get_fast_api_app(
    agents_dir=agents_dir,
    session_service_uri="postgresql+asyncpg://...",   # sessions in Postgres
    memory_service_uri=MEMORY_SERVICE_URI,             # long-term memory too
    session_db_kwargs={
        "pool_size": 3, "max_overflow": 2,              # 5 connections per worker
        "pool_recycle": 300, "pool_pre_ping": True,
    },
)

Now every session and every event is a row in the database. A pod can restart ten times a day; the conversation is still there when it comes back. And as a bonus, every conversation is now auditable, which matters a lot once real businesses run on it.

We did the same for long-term memory (what an agent remembers across conversations). The default there was also in RAM, per worker, and never fed. It now lives in the application database, scoped by agent and by user, so one user's memories never leak into another's.

Two small details that only show up in production:

Mind the connection pool. Out of the box, the session engine would open up to 15 connections per worker (SQLAlchemy's default 5 plus 10 overflow). On a database shared with other services, that adds up fast. We capped it at 5.
Memory grows forever if you let it. ADK's database session service has no retention policy: the events table gets one row per agent turn, each holding the full event as JSON. Left alone, it slowly eats the database. A small background task now deletes events older than 90 days, once a day. Sessions keep their shell; they just lose old turn details.

Fix 2: take the orchestrator out of the browser

The workflow engine moved to the server, built on ADK's own Workflow primitives (chaining, fan-out, join) instead of a home-made scheduler. Every step emits step_start, step_complete or step_error, and the run's outcome is written to the database in a finally block.

To be precise about what changed: closing the tab still stops the run. That's deliberate. A client that disconnects cancels its run, and the cancellation is propagated to everything the workflow started (sub-workflows, retry waits), so nothing keeps running unsupervised in the background. What changed is that the run no longer vanishes. It's recorded as cancelled, with its input, and a cancelled run can be replayed. If it's safe, that is. Hold that thought until Part 2.

Fix 3: write every run down before it starts

This is the one that made everything else possible. Every run (a workflow, a scheduled agent, a chat turn) now gets one row, written the moment it starts. Incoming webhooks keep their own durable queue, with its own idempotency; the two will merge the day the execution engine is unified.

Here's what that row holds (run_store.py):

Table("agent_runs", metadata,
    Column("run_id", String, primary_key=True),
    Column("trigger", String),        # workflow | schedule | chat | webhook
    Column("owner_id", String),
    Column("config", String),         # the input. No input, no replay.
    Column("input_file_path", String),
    Column("status", String),         # running | success | error | cancelled
    Column("attempts", Integer),
    Column("error_message", String),
    Column("replay_of", String),      # the run this one replays
    Column("tools_executed", String), # more on this one below
)

If the run came with a file, we copy that file to disk right away. That one came from experience: the email path already kept attachments, precisely so we could replay an email that had since been deleted from the mailbox. If you only keep a pointer to the user's file, the day it disappears, your replay is impossible.

Fix 4: memory on a diet

Persisting memory solved amnesia. It also created a new problem: an agent that remembers everything can choke on it.

ADK sends the full conversation history to the model on every turn. Now imagine a tool that converts a PDF into images so the model can read it. Each page is a base64 blob worth roughly 100k to 150k tokens. After five turns, the model has been sent the same image five times. You're paying, five times, for something it already read.

So before every model call, a callback strips large base64 payloads from every message except the latest one:

def before_model_callback(*, callback_context, llm_request):
    contents = llm_request.contents or []
    for content in contents[:-1]:            # never touch the latest turn
        for part in content.parts or []:
            fr = getattr(part, "function_response", None)
            if fr and fr.response:
                _strip_payload(fr.response)  # big base64 -> "<stripped: N chars>"
    return None

The model already looked at the image when it arrived. From the next turn on, its own text about the image is the source of truth. It still sees that the tool ran and what it produced, just not the raw bytes again.

Amnesia: solved. Our agents now remember. Which brings us to the second, sneakier problem.

Part 2: Déjà vu

The tempting fix

Once runs are written down with their input, the obvious next idea is: when a run fails, just relaunch it. Same input, same agent, go again. Fully automatic. Problem solved, right?

Picture an agent whose job is to process an incoming customer request. It reads the message, drafts a reply, sends the email, then updates the CRM. Halfway through the CRM update, the pod gets evicted.

Relaunch it from the start, and it reads the message, drafts a reply, and sends the email again. The customer now has two answers. Maybe two slightly different answers, because the model doesn't write the same thing twice.

That's the core of the problem. An agent isn't a pure function. It acts on the world through its tools: it sends emails, writes to databases, calls APIs. Those are side effects. And a blind relaunch repeats every one of them.

Why you can't just "resume from step 3"

The natural reaction is: fine, then don't restart from the beginning, resume from where it stopped. Skip the steps that are already done.

With a classic workflow engine, that works, because steps are fixed and deterministic. Step 3 is always step 3.

With an LLM agent, there is no step 3. The "plan" only exists inside that particular run, in the model's reasoning at that moment. Run it again and the model may take a different path, call the tools in a different order, or decide it needs a tool it didn't use the first time. There's no clean way to replay an agent while skipping the calls it already made, because you can't replay the reasoning that led to them.

So we stopped trying to answer "how do we resume?" and asked a different question:

How do we know whether it's safe to start again?

To answer that, you need exactly one piece of information: did this run already touch the world?

Fix 5: record every tool call (the call, not the answer)

Every agent run now records the list of tools it called. We read it straight from the session events (run_main.py):

def executed_tools(events) -> list[str]:
    """Tools called in a list of ADK events, in order.

    The call (functionCall) is what counts, not the response:
    a tool that crashes midway may have acted without ever answering.
    """
    names = []
    for event in events:
        for part in (event.get("content") or {}).get("parts") or []:
            call = part.get("functionCall") or part.get("function_call")
            if call and call.get("name"):
                names.append(call["name"])
    return names

That docstring hides the most important detail in this whole post. We count the call, not the response. If the email tool sends the email and then times out before answering, there's no response in the history. But the email is gone. If you only counted tools that answered, you'd conclude "nothing happened" and send it again.

Fix 6: the memory is what makes this possible after a crash

Here's where the two halves of this post connect.

When a run fails with an exception, its response is lost. We don't get the nice list of events back. But the session is in Postgres now. It kept everything that happened before the crash. So after a failure, we go read the session itself and extract the tool calls from it.

One subtlety: a chat session contains the whole conversation, including tools called in previous turns. Those aren't side effects of this run. So we only look at what happened after the user's last message.

Without Fix 1, this is impossible. Amnesia doesn't just lose the conversation, it destroys the evidence you need to decide whether relaunching is safe.

Fix 7: "unknown" is not "none"

The tools_executed column has three possible states, and the difference between two of them is the whole safety model:

[]: we checked, and no tool was called. Safe to relaunch.
["send_email", "update_crm"]: we checked, and these were called. Relaunching repeats them.
NULL: we couldn't check (the session was unreadable, for instance). We don't know.

It's very tempting to treat "unknown" as "probably nothing". We treat it as "probably something", because the cost of being wrong isn't symmetric. Refusing a safe relaunch costs a click. Approving an unsafe one costs a duplicate email to a customer.

Fix 8: the replay rules

With all of that in place, every replay goes through a single endpoint, POST /runs/{run_id}/replay, and the rules for replaying an agent run (a scheduled agent or a chat turn) fit in a table:

The original run...	What happens when you ask to replay it
succeeded	Refused. Its side effects already happened.
is still running	Refused. Wait for it to settle.
failed, and we know no tool was called	Replayed. Nothing to repeat.
failed after calling tools	Refused, with the list of tools it called.
failed, and we can't tell	Refused. Unknown is not none.

And the refusals aren't silent. They're an HTTP 409 with a message a human can act on:

def _refuse_if_side_effects(tools_executed: str | None) -> None:
    if tools_executed is None:
        raise HTTPException(409,
            "It is unknown whether this run called any tool before failing, "
            "so a replay could repeat a side effect. "
            "Pass force=true to run it again anyway.")
    tools = json.loads(tools_executed or "[]")
    if tools:
        raise HTTPException(409,
            f"This run already called {', '.join(tools)} before failing: "
            "a replay would call them again. "
            "Pass force=true to run it again anyway.")

force=true exists, on purpose. Sometimes repeating the action is fine, or even what you want. But that's a decision for a human who has read the message, not for an automatic retry loop at 3 a.m.

Three more rules that matter:

A replay is a new run that cites the original (replay_of). We never overwrite history. If the replay fails too, you can tell it apart from the original failure.
A replay starts in a fresh session. The original session may contain a half-finished exchange, and for chat it's the user's own conversation, where a replay has no business writing.
We never replay "almost". If the input file has disappeared from disk, the replay is refused (410 Gone). Relaunching without it would produce a run that silently differs from the one you think you're reproducing.

Fix 9: runs that die without saying goodbye

One last trap. A run marks itself as finished in a finally block. But when Kubernetes kills a pod, the finally never runs. The row stays running. Forever. And a run that's "still running" can never be replayed.

So whenever we read a run, we reconcile it: if it's been running for more than six hours with no completion recorded, we mark it error with an explicit message, "no completion was recorded within 6h; its actual outcome is unknown":

conn.execute(
    run_table.update()
    .where(run_table.c.run_id == run_id,
           run_table.c.status == "running")   # only if still running
    .values(status="error", error_message=message, finished_at=now())
)

The WHERE status = 'running' makes it safe even when several replicas reconcile the same run at the same time: only the first write lands. Six hours is deliberately generous, so we never close a genuinely long run by mistake. And once reconciled, the run goes through exactly the same safety rules as any other failure. If it had already sent that email, it won't send it again.

What does get retried automatically

None of this means nothing is automatic. Things are retried silently when, and only when, it's provably harmless:

Empty model turns. Sometimes the model returns an empty turn (it happens, especially with "thinking" models). By definition an empty turn contains no tool call, so asking again can't repeat a side effect. We retry it once before falling back.
Event-triggered runs. Incoming webhooks go through a durable queue: each event is processed once, a crashed event is picked up again by another worker, and retries back off and stop after a ceiling, so nothing spins forever.

One honest footnote on the empty-turn retry: ADK appends the user's message to the session on every request, so a retry leaves that message twice in the history. It's rare and harmless today, but it's on our watch list for very long sessions.

The mental model

If you remember one thing from this post, make it these three questions. Ask them before your agent relaunches anything:

Do I remember? Is the agent's memory (sessions, events, inputs) stored somewhere that survives a restart?
Do I know what it already did? Do I record every tool call, not just every tool answer?
Is doing it again harmless? And if I can't tell, do I refuse, or do I gamble?

Amnesia is a "no" to the first question. Déjà vu is skipping the other two.

What this doesn't solve (yet)

What we have today is safe replay: a failed run is either relaunched because we know it's harmless, or refused with an explanation. What we don't have yet is true resume: picking up a run mid-flight, reusing the results of tools that already ran instead of calling them again.

Getting there probably means making tools themselves idempotent (an idempotency key on every side-effecting call, so "send this email" twice is still one email) and caching tool results per run. That's the next frontier, and it's a hard one. If you've built it cleanly, I'd genuinely love to read about it.

Is this overkill?

It's mostly one table (agent_runs), one setting (session_service_uri), one callback, and a handful of rules. No new infrastructure.

And it's what lets our agents run in production for months, across industrial use cases, on Kubernetes with our Helm chart and fully on-prem with Docker Compose, restarting several times a day without anyone noticing. Which is exactly how it should be.

Try it yourself

The whole stack, database included, comes up with three commands from the apowerb-hosting repository:

git clone https://github.com/apowerb/apowerb-hosting.git && cd apowerb-hosting
cp .env.example .env && ./scripts/generate-secrets.sh
docker compose -f docker-compose/docker-compose.yml --env-file .env up -d

The interface is on http://localhost:3000 and the API on port 8000. Add a model key in the interface and you can start breaking agents on purpose: kill the container mid-run, restart it, and watch the session come back.

Going to Kubernetes? The same repository holds the Helm chart (also on Artifact Hub) and plain Kubernetes manifests. The Docker Compose guide and the Kubernetes guide walk through both, and I wrote up the honest version of the Helm install, traps included.

Before you let your agent relaunch itself, check this

[ ] Are sessions stored somewhere other than RAM?
[ ] Is long-term memory scoped per agent and per user?
[ ] Is every run written down, with its input, before it starts?
[ ] Are input files copied, not just referenced?
[ ] Do you record tool calls, including those that never answered?
[ ] Do you treat "unknown" as "unsafe"?
[ ] Is a replay a new run that cites the original, in a fresh session?
[ ] Do runs killed mid-flight get reconciled instead of staying "running" forever?
[ ] Are old events purged, and old payloads stripped from the context?

What I'd tell myself before we shipped

Defaults are written for demos. In-memory sessions are perfect until your first restart. Check every default that stores state.
Never trust "nothing happened" unless you checked. Unknown is not none.
A refusal with a clear message beats a clever automatic retry. Your users will forgive a button that says "this already sent an email, are you sure?". They won't forgive the second email.

Come break it with us

All of this lives in ApowerB, our open-source agent runtime (Apache 2.0). The replay logic is in run_main.py, the run table in run_store.py, and the memory diet in history_compaction.py. The comments explain the scars; go read them, and tell us where we got it wrong.

Issues and PRs are welcome. If you want a first contribution in this area, open an issue or pick an existing one, and I'm happy to pair on your first PR. And if this post was useful, a star on the repo helps us a lot.

Source code: github.com/apowerb/apowerb
Self-hosting (Docker Compose, Helm chart, Kubernetes manifests): github.com/apowerb/apowerb-hosting
Helm chart on Artifact Hub: artifacthub.io/packages/helm/apowerb/apowerb-chart
Docker image: hub.docker.com/r/apowerb/apowerb
Documentation: docs.apowerb.com
Product page: thaink2.com/apowerb
See an agent run end to end (75s): youtu.be/L3KoK0gqekY

Now I'm curious: what does your agent do when it crashes halfway through a task?
Start over, resume, give up? Tell me in the comments, I read every one.

Deploying an open-source AI agent platform to Kubernetes: the honest one-command version

Anis Meziani — Mon, 05 Oct 2026 14:13:00 +0000

Every "deploy X to Kubernetes in one command" article is technically true and practically a lie. The helm install really is one line. What the post skips is the empty cluster with no storage class, no ingress controller, no cert-manager, and no DNS. That 90% is what stands between you and a working URL. So here's the honest version for ApowerB, the open-source agent platform: the chart genuinely installs in one command, and here is everything around it, traps included.

What the chart actually deploys

ApowerB ships an official Helm chart (apowerb-chart, OCI, cosign-signed) that stands up the whole stack, not just the app:

Backend and Next.js frontend: the agent runtime and its UI
PostgreSQL: where your agents live (they're rows, materialized into executable modules at load time)
th2etl: pipeline and orchestration, with a seed job
th2pulse: log and trace storage
otel-collector: telemetry routing
th2forecast: optional forecasting engine (disabled by default)
A PersistentVolumeClaim for uploads, RAG artifacts and agent pools

That's a platform, not a container. Which is exactly why the cluster prerequisites below matter more than for a toy app.

The fast path (local or dev cluster)

If you already have a cluster with a working storage class, two commands get you running.

1. Generate the secrets. The chart refuses to start without them, on purpose:

umask 077
cat > values-secrets.yaml <<YAML
backend:
  env:
    encryptKey: "$(openssl rand -base64 32)"
th2etl:
  apiKey: "$(openssl rand -hex 32)"
th2pulse:
  ingestToken: "$(openssl rand -hex 32)"
  queryToken: "$(openssl rand -hex 32)"
postgres:
  password: "$(openssl rand -hex 16)"
YAML
chmod 600 values-secrets.yaml

2. Install:

helm upgrade --install apowerb \
  oci://registry-1.docker.io/apowerb/apowerb-chart --version 0.4.23 \
  --namespace apowerb --create-namespace \
  --values values-secrets.yaml --timeout 10m

Then port-forward and open the UI:

kubectl -n apowerb port-forward svc/apowerb-frontend 13000:3000 &
# http://localhost:13000

Add a model key in the UI and you have a working agent platform on your cluster. That's the "one command" part, and it's real.

The one rule that bites everyone: pass --values values-secrets.yaml on every helm upgrade. Drop it and you rotate your own credentials out from under a running database. Keep that file safe and version it in your secret store, not in Git.

The honest production path (empty cluster to HTTPS URL)

A real cluster rarely has everything wired. Here's the full walk, which is where most tutorials go quiet.

Prerequisites

Kubernetes 1.28+. Three nodes at 4 vCPU / 8 GB give comfortable headroom. Check the ground first:

kubectl get nodes          # all Ready
kubectl get storageclass   # must not be empty
kubectl get ingressclass   # must exist

Step 1: Storage that actually binds

Lean managed clusters often ship with no default storage class. If get storageclass is empty, install one:

curl -sL -o local-path-storage.yaml \
  https://raw.githubusercontent.com/rancher/local-path-provisioner/v0.0.37/deploy/local-path-storage.yaml
kubectl apply -f local-path-storage.yaml
kubectl -n local-path-storage rollout status deploy/local-path-provisioner --timeout=120s

kubectl patch storageclass local-path \
  -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

Trap: most storage classes are WaitForFirstConsumer, so a lone PVC stays Pending and looks broken when it's fine. Test with a PVC and a pod together:

kubectl apply -f - <<'YAML'
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: probe-pvc}
spec:
  accessModes: ["ReadWriteOnce"]
  resources: {requests: {storage: 1Gi}}
---
apiVersion: v1
kind: Pod
metadata: {name: probe-pod}
spec:
  restartPolicy: Never
  containers:
    - name: writer
      image: busybox:1.37
      command: ["sh","-c","echo ok > /data/marker && cat /data/marker && sleep 20"]
      volumeMounts: [{name: data, mountPath: /data}]
  volumes:
    - {name: data, persistentVolumeClaim: {claimName: probe-pvc}}
YAML
sleep 25 && kubectl get pvc probe-pvc && kubectl logs probe-pod
kubectl delete pvc probe-pvc pod probe-pod

Want Bound and ok in the logs. If this fails, stop. Nothing else will work.

Step 2: Ingress

kubectl get ingressclass
kubectl get svc -A | grep LoadBalancer

Note the class name (Traefik and ingress-nginx are the usual managed add-ons) and the EXTERNAL-IP for DNS.

Step 3: cert-manager and a staging issuer

Let's Encrypt rate-limits at 5 failures/hour, so always start on staging:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: you@example.com
    privateKeySecretRef:
      name: letsencrypt-staging-account-key
    solvers:
      - http01:
          ingress:
            ingressClassName: traefik   # your actual class from Step 2

Trap that costs an afternoon: use ingressClassName, not the deprecated class annotation. Otherwise the ACME challenge Ingress gets no recognised class and the certificate sits at Ready: False forever.

Step 4: Deploy with ingress and TLS

Same install command, now with public access turned on:

helm upgrade --install apowerb \
  oci://registry-1.docker.io/apowerb/apowerb-chart --version 0.4.23 \
  --namespace apowerb --values values-secrets.yaml \
  --set ingress.enabled=true \
  --set ingress.className=traefik \
  --set ingress.host=apowerb.example.com \
  --set ingress.tlsEnabled=true \
  --set ingress.annotations."cert-manager\.io/cluster-issuer"=letsencrypt-staging

Create the first admin while you're at it:

  --set superadmin.email=admin@example.com \
  --set superadmin.password='<a strong one>'

Step 5: DNS and verify

Point an A record at the EXTERNAL-IP, then:

kubectl -n apowerb get certificate
kubectl -n apowerb describe certificate apowerb-tls

Stuck at Ready: False? It's almost always one of: DNS not resolving to the load balancer, port 80 closed (HTTP-01 needs it), the class vs ingressClassName trap, or a staging rate limit. Once it's True, swap the issuer to production (https://acme-v02.api.letsencrypt.org/directory) and reapply.

Configuring the platform

The same values mechanism covers the rest:

Default LLM: defaultLlm.model and defaultLlm.apiKey for agents without their own key (LiteLLM under the hood, so OpenAI / Anthropic / Mistral / Gemini / local all work)
Integrations: GOOGLE_INTEGRATION_* (Drive, Gmail, Calendar), MICROSOFT_INTEGRATION_* (Outlook, webhooks, mail), SMTP_*
Storage: storage.mode and S3_* to push bi_store, uploads, artifacts_store and agents_pool to object storage instead of the PVC

Verdict

The install itself is genuinely one command against an OCI chart, and the chart is honest: it won't boot without real secrets, and it bundles the orchestration, log store and telemetry rather than leaving them as "exercises for the reader." The work that remains isn't ApowerB's, it's the cluster's: storage, ingress, cert-manager, DNS. And that's true of anything serious you deploy.

If you want to see the easy version first, the Docker Compose stack comes up locally in a few commands. But if you're putting agents in front of real users, the Helm chart is the path, and now you've seen all of it, traps included.

Your agent framework stops at the demo. This open-source stack ships to production.

Anis Meziani — Thu, 24 Sep 2026 12:47:05 +0000

Building an AI agent is easy. Building the system around it is where things get complicated.

You can create an agent in a few lines of Python. Then you need an API, persistence, authentication, tool integrations, RAG, scheduling, observability, a UI, deployment, and a way to operate the whole thing. That's the gap I wanted to explore when I started digging into apowerb, an open-source AI agent framework from thaink².

Rather than treating it as another agent SDK, I wanted to answer a more specific question:

What does an AI agent runtime actually need to provide once the prototype becomes a real system?

This article walks through apowerb's architecture, its runtime model, RAG engine, Text-to-SQL support, event-driven execution, deployment model, and how it compares with other popular approaches.

apowerb / apowerb

The open-source agentic framework to build, orchestrate, and operate production AI agents.

Build, run and govern AI agents that work while you don't.

The open-source agentic framework to build, orchestrate, and operate production AI agents.

Documentation • Quickstart • API Reference • Deployment • thaink2

apowerb is an open-source platform to build, run and govern AI agents in production. It covers multi-agent orchestration, RAG and Text-to-SQL, MCP servers and built-in business tools, email- and schedule-triggered runs, token quotas and session audit. You can self-host it with Docker Compose or Helm. The core is Apache-2.0; a few enterprise modules (billing, identity-provider sign-in, MFA agent evaluation, the supervision screen) are sold separately by thaink².

Build agents from the web interface or the REST API — instructions, model tools, knowledge bases — and compose them: sequential, parallel, loop, sub-agents.
Connect them to your systems: any MCP server, plus built-in tools for Google Workspace, Microsoft 365, GitHub, HubSpot, Pipedrive, Odoo, SQL databases Oracle, MongoDB, S3,…

View on GitHub

What is apowerb, exactly?

Most agent frameworks start from a library. You import the framework, define an agent, give it tools, connect an LLM, and build your application. That model is useful, but it leaves a lot of infrastructure outside the framework: API layer, persistence, authentication, UI, deployment, scheduling, event handling, observability, RAG, integrations.

apowerb takes a different approach. Instead of treating the agent as an object embedded inside an application, it treats the agent as something that can be defined, stored, materialized and operated by a runtime. The stack:

FastAPI : REST APIs and Server-Sent Events
Typer : the CLI (apowerb serve, apowerb agents list, …)
Next.js : the web UI (build agents, give them tools/knowledge, run and inspect them)
PostgreSQL : agent and configuration state
Google ADK : the execution engine
LiteLLM : the model routing layer
Docker Compose / Helm : deployment

The result is closer to an application runtime than to a standalone Python library.

The architecture in one picture

┌────────────────────────────────────────────────────────┐
│                     Clients / UI                        │
│    Next.js UI  •  Typer CLI  •  REST API  •  SSE         │
└──────────────────────────┬─────────────────────────────┘
                           │
                           ▼
┌────────────────────────────────────────────────────────┐
│                 FastAPI Orchestrator                    │
│   Agent materialization   •   Webhooks (Pub/Sub)        │
│   Cron / event scheduling •   SSE output stream         │
└──────────────────────────┬─────────────────────────────┘
                           │
                           ▼
┌────────────────────────────────────────────────────────┐
│              Execution Engine Google ADK              │
│   Base • Sequential • Parallel • Loop                   │
│   Hierarchical sub-agents  •  Human-in-the-loop         │
└─────────────┬───────────────────────────┬──────────────┘
              │                           │
              ▼                           ▼
┌─────────────────────────┐    ┌─────────────────────────┐
│   LiteLLM Model Router  │    │     Tools & Services     │
│ OpenAI • Anthropic      │    │ M365 • Google • SQL      │
│ Gemini • Mistral • local│    │ RAG • Webhooks           │
└─────────────────────────┘    └─────────────────────────┘

Four orchestration patterns are built into the runtime:

base : single-turn conversational agents with tool execution
sequential : deterministic step-by-step pipelines passing context downstream
parallel : concurrent branches aggregated for synthesis
loop : evaluator/optimizer cycles that iterate until a target condition or stop token

Hierarchical sub-agents let a coordinator delegate work to specialized agents. The interesting part, though, isn't the list of patterns, it's what happens before execution.

The interesting choice: agents as runtime objects

In a typical application, an agent is defined directly in Python and the application owns that definition:

agent = SomeAgent(
    model="...",
    tools=[...],
    instructions="...",
)

apowerb takes another route. Agent definitions, tools, skills and their configuration live in PostgreSQL. At startup, the backend uses those definitions to materialize executable Python modules.

Agent definition ─► PostgreSQL ─► Materialization ─► Python module ─► Google ADK
                                                                        ├─► Tools
                                                                        ├─► RAG
                                                                        └─► LLM

That changes the operational model. The application isn't rebuilt every time the agent catalogue changes; the runtime loads definitions from the database and turns them into executable agents. For an organisation managing many agents, that's a meaningful distinction, the agent becomes closer to a runtime-managed resource than a static piece of application code. It's the detail I found most interesting in the project.

Model access through LiteLLM

The execution layer is deliberately separated from model access. apowerb uses LiteLLM as the routing layer, so the runtime can work with multiple providers without coupling an agent's tools to one vendor. The documented stack includes Anthropic, OpenAI, Mistral, Gemini, OVHcloud, Groq, and local Ollama/vLLM deployments.

Agent ─► Google ADK ─► LiteLLM ─┬─► OpenAI
                                ├─► Anthropic
                                └─► Local (Ollama / vLLM)

The tools and orchestration logic don't need to know which provider ultimately executes the model call, useful for teams that want to swap or benchmark models.

RAG is part of the stack

RAG isn't treated as an external SaaS dependency. apowerb includes th2rag, a standalone FastAPI service for ingestion and retrieval:

Document ─► Conversion (Docling) ─► Chunking ─► Embeddings ─► LanceDB ─► Retrieval ─► Generation

The current implementation uses Docling for document conversion, its HybridChunker for semantic chunking, sentence-transformers/gtr-t5-large for embeddings, LanceDB as the vector store, a LanceDB retriever for similarity search, and Mistral AI for generation. Each stage sits behind a base class (BaseEmbedding, BaseGenerator, …), so swapping the embedding model or the LLM is a matter of subclassing, not surgery.

The service can index local files, URLs, database query results and S3 objects, into knowledge bases associated with individual agents rather than one global index. That makes RAG a runtime capability instead of something each application assembles separately.

Native Text-to-SQL

An agent can connect to PostgreSQL or MySQL, introspect the schema, generate SQL from a natural-language question, execute it and return the result:

"Which customers generated the most revenue last quarter?"
        │
        ▼
   Agent / LLM ─► Schema context ─► SQL generation ─► Query ─► Results ─► NL response

The interesting part isn't that an LLM can write SQL, it's that the connection, schema introspection, generation and execution all happen inside the agent runtime. Text-to-SQL becomes another runtime capability rather than an integration every application builds from scratch.

Event-driven agents

Most agent demos are request/response:

HTTP request ─► Agent ─► Response

apowerb can also work in the opposite direction:

External event ─► Webhook ─► Agent ─► Tools / RAG / LLM ─► Action / Result

The current implementation includes email-triggered agents. Gmail uses Google Cloud Pub/Sub push notifications; Outlook uses Microsoft Graph change subscriptions with clientState validation.

Email arrives ─► Provider notification ─► apowerb webhook ─► Fetch message ─► Run agent ─► Log result

Subscription renewal is handled in the background: Gmail watches expire after seven days, Outlook after three, and the runtime periodically renews anything approaching expiration. Combined with cron-style scheduled runs and SSE streaming (token-by-token output, RAG ingestion progress, notifications), this moves the runtime beyond chatbots, the agent becomes a component in an event-driven system.

The open-source boundary

The core stack is Apache 2.0 ; the framework plus supporting services like the RAG engine, ETL components, observability and web UI. The project follows an open-core model: some commercial extensions (billing, MFA, identity-provider SSO, agent evaluation, supervision, organisation management) are maintained separately.

A TH2_EXTENSIONS environment variable declares which commercial bricks to load; if a brick isn't installed or enabled, the core doesn't depend on it. Routes belonging to an absent brick return a clean 404. The useful distinction: the open-source runtime remains fully usable without the commercial extensions, the core isn't an artificially limited demo.

It's more than one repository

A common pattern in AI tooling is "open-source framework + several proprietary services around it." Here, much more of the surrounding infrastructure is exposed. The thaink² GitHub org holds 10 repositories, all Apache 2.0:

Repository	Role	Technology
`apowerb`	FastAPI backend, CLI and ADK runtime	Python
`apowerb-ui`	Visual agent builder and monitoring	Next.js
`th2rag`	RAG, parsing, embeddings, vector search	LanceDB / Docling / Mistral
`th2etl`	ETL pipelines	Python
`th2pulse`	Observability and logs	OpenTelemetry
`th2forecast`	Time-series forecasting	R
`apowerb-hosting`	Docker Compose / Kubernetes / Helm	—
`apowerb-docs`	Documentation	Mintlify / MDX
`agent-hub`	Community agent templates	—

You can read and run the RAG engine, the scheduler and the telemetry pipeline, there's no hidden proprietary service doing the heavy lifting behind an API.

Run the full stack

The fastest way to understand the project is to run it. The hosting repository provides a Docker Compose setup:

git clone https://github.com/apowerb/apowerb-hosting.git
cd apowerb-hosting

cp .env.example .env
./scripts/generate-secrets.sh

docker compose \
  -f docker-compose/docker-compose.yml \
  --env-file .env \
  up -d

The UI comes up on http://localhost:3000, the API on :8000, and PostgreSQL runs inside the stack. You still provide an LLM API key, through the UI, or via DEFAULT_LLM_MODEL / DEFAULT_LLM_API_KEY. If you only want the Python package:

uv add apowerb   # or: pip install apowerb

Two entry points, two purposes:

apowerb package   ─► embed the runtime in your application
apowerb-hosting   ─► run the complete platform

How it compares

Comparisons between agent frameworks are easy to get wrong because they solve different problems. The useful question isn't "which framework is best?" but "which layer does each project expect you to own?" Below, "built-in" means the capability is part of the platform rather than something you assemble around it.

Capability	apowerb	LangGraph	CrewAI	n8n + LLM
Multi-model	LiteLLM	Yes	Yes	Partial
Built-in RAG	th2rag	Yes	Partial	Partial
Email webhooks	Gmail + Outlook	No	No	Yes
Native Text-to-SQL	Yes	Via tools	No	Partial
Included UI	Next.js	No	No	Yes
Helm deployment	Yes	DIY	DIY	Yes
Main approach	Runtime / platform	Framework	Framework	Workflow automation

The distinction isn't one checkbox, it's the architecture. LangGraph and CrewAI are frameworks you embed into an application. n8n comes from workflow automation, where an LLM is one node in a larger graph. apowerb provides the runtime around the agent: API, persistence, UI, integrations, scheduling, RAG and deployment. Different design choices, not just different implementations of the same product.

What I found interesting

1. Agents are treated as managed runtime resources. The database-backed definition and materialization model is the most distinctive architectural choice.

2. RAG isn't an external dependency. The project ships its own RAG service instead of requiring every deployment to assemble one.

3. Event-driven execution is first-class. Gmail, Outlook, schedules and SSE make "proactive" agents implementable, not just request/response chatbots.

Trade-offs

Not everything is settled. The release cadence is high, so the API isn't frozen, expect to track changes if you build on it now. Schema is created by ensure_* functions at startup rather than Alembic, which is fine today but means no reversible migrations. And the external community is still small; most contribution looks internal so far, so you'd be an early adopter.

None of that is a dealbreaker, it's the normal shape of a young project that's visibly running in production.

Worth a clone if…

you want RAG, tool auth, an API and a UI to already exist so you can build the actual agent;
you're a data team and Text-to-SQL over your warehouse plus scheduled runs maps to your use case;
you need everything self-hosted and auditable, your models, your data, Apache 2.0;
you're benchmarking agent frameworks and want to see where the "runtime, not library" line falls.

Three experiments that take minutes each: point an agent at a Postgres and query it in English; drop a folder of PDFs into a knowledge base and ask questions; set a Gmail trigger that summarizes and routes incoming mail.