DEV Community: Tisha Chawla

Your Agent Has a 1987 Database Problem

Tisha Chawla — Mon, 21 Sep 2026 09:53:51 +0000

Here is the shape of a bug I have watched people chase for days.

14:22:03.204  tool    POST /v1/charges              201 Created
14:22:03.209  db      COMMIT orders/8812 status=paid
14:22:03.211  x       process killed (out of memory)
14:22:11.470  worker  resume th_44c1 from ckpt_19
14:22:11.882  graph   node=tools    call_7f3a has no result
14:22:11.955  tool    POST /v1/charges              201 Created

The customer was charged twice, and nothing in that trace failed. No exception, no retry counter, no red span in the viewer. Both charges returned clean 201s, and the agent did exactly what it was built to do.

So why did it charge twice? The answer sits one layer down, in how an agent loop records what it has already done.

How an agent loop actually works

A model has no memory. Calling one is a pure function call. It does not remember the previous call, holds no connection to your database, and has no idea whether it charged someone thirty seconds ago. Everything it knows arrives as input, and the moment the call returns it knows nothing again.

So the agent's memory is a list of messages your runtime owns. The system prompt, the user's message, each assistant reply, each tool requested, each result handed back. That list lives in Postgres and gets passed to the model on every turn. When people say an agent knows something, this list is what they mean, because there is nowhere else for knowledge to live.

One turn contains two separate writes. Your code executes the tool, which moves money at a payment provider and writes a row in your own database. Then it appends the result to the list, and the runtime saves the list. Executing and saving happen at different moments.

In the gap between them, the agent's memory disagrees with the world. The charge exists permanently at the provider. The list does not mention it.

The rest of this article is about that gap.

Whether it matters to you depends on what your tools actually do. An agent that searches, summarizes or drafts can crash and replay all day without harm, because running a read a second time costs latency and nothing else. The gap only bites when a tool changes something outside your process that you cannot take back, meaning money moving, an email going out, a ticket filed, a row written into somebody else's system. If your agent only reads, none of what follows is urgent. If it acts, all of it is.

What happened in the trace

Two questions to answer: where was the gap, and what was in the checkpoint when the process died?

LangGraph runs your agent as a graph. One node calls the model, a second node executes whatever tools the model asked for, and after each of them a component called the checkpointer saves the message list.

The checkpointer is the save function, and nothing more exotic than that. It writes the agent's current state to storage, keyed by conversation, so a different process can pick the run up later. In development it is usually an in-memory stub; in production it is Postgres. Each save it makes is a checkpoint, and ckpt_19 in the trace is one of them.

At ckpt_19, that saved list ended like this:

assistant   "I'll charge the customer."
            tool_calls: [ charge_customer(cus_8812, 4900), id=call_7f3a ]

That is a question with no answer: a tool was requested, nothing has come back, and the graph's next move is to run the tools node.

The tools node then did four things in order:

Called the payment provider.
Took the 201 back.
Committed the order row.
Built the reply message that would close out call_7f3a.

It was killed before it handed that reply back. Steps one and three are permanent and outside your control now, because the money moved and the order row committed. Steps two and four existed only in the memory of a process that no longer exists.

So the checkpointer was never given anything to save, and ckpt_19 still ends exactly as it did above, with a tool call and no result.

When the worker resumed eight seconds later it loaded that list and did the only thing available to it. An unanswered tool call means run the tool, so it ran the tool.

Nothing malfunctioned. The graph followed its own rules against a record that was missing an entry, and in LangGraph this is documented behavior rather than a bug, because saving the list is a step that comes after the tool has already done its work.

So the question is not why the agent repeated itself, since given that list it had no other option. The question is why the charge and the record of the charge were allowed to come apart at all. Two things needed to happen together and did not, which is a problem databases have been solving since 1978.

1978: all or nothing

Move fifty dollars between two accounts and you have two writes: subtract from one, add to the other. If the machine dies in between, the money is not queued somewhere, it is gone, and nothing in the system knows it existed. The transaction had to be invented to stop this. Until it existed, an application that needed two writes to hold together had to track that itself and hope the machine stayed up, which is why banks reconciled their books overnight and fixed the discrepancies in the morning.

The fix is not the one people assume. You cannot make two writes happen simultaneously, so databases stopped trying and did something else: they write down what they are about to do before doing any of it.

That record goes into a log, a file the database only ever appends to. It is flushed to disk, meaning the operating system is forced to put the bytes on the disk itself rather than hold them in memory where a power cut would erase them. Only then does the database touch the actual rows.

For the fifty dollar transfer, the log holds roughly this:

txn 41  begin
txn 41  account A   100 -> 50
txn 41  account B    20 -> 70
txn 41  commit

The last line decides everything. On restart the database reads the log. If the commit line is there, it applies every change above it. If the commit line is missing, it undoes them. No judgment is involved. One line either reached the disk or it did not.

That is write-ahead logging. Jim Gray described it in Notes on Data Base Operating Systems in 1978, and in 1983 Härder and Reuter named the guarantees it provides: ACID, whose A is atomicity.

Atomicity is not what the word suggests. The two writes never happen at the same instant. They happen one after another, and the machine can die between them. What makes them atomic is that one commit line. It was written before either change was applied, and it decides afterwards whether both count or neither does.

That only works because the database owns both accounts. They are rows in its own files, covered by its own log, so a single commit line can settle both at once.

A charge at a payment provider is not a row in your database. Your log cannot describe it, your commit line cannot settle it, your rollback cannot undo it. Nothing marks that edge in your code either: no error, no type mismatch, the two calls look the same.

So you need the 1978 trick without the database doing it for you. Somewhere you control, write down "I am about to charge this card" before you charge it, so a crash leaves a note behind and the next process can work out what happened.

1978 to 1991: trying to leave the box

One database's log can promise nothing about what happened in another, and the obvious move is a bigger transaction spanning both.

That runs into an obstacle nobody has removed since. Akkoyunlu, Ekanadham and Huber published the impossibility proof in 1975, and Gray named it the Two Generals paradox in that same 1978 paper: you send a request, it times out, and you cannot distinguish "it never arrived" from "it arrived, ran, and the acknowledgment was lost coming back." Those two worlds look identical from where you stand, and they still look identical after you add retries, better timeouts or a service mesh. The ambiguity is not a gap in your tooling.

The field built two-phase commit anyway. A coordinator asks every participant to promise it can commit, waits for all of them to agree, then tells everyone to proceed. Standardized as X/Open XA in 1991, it delivers genuine atomicity across systems.

The cost is availability. To keep its promise, each participant has to lock the rows involved, meaning hold them so that nobody else can read or change them until the outcome is settled. If the coordinator dies between the promise and the instruction to commit, every participant sits holding those locks and waiting for orders that are not coming. It also requires every participant to implement the protocol, which looks harmless in 1991 and proves fatal twenty years later.

Two-phase commit also assumes the work is short enough to hold locks through. What if it isn't?

1987: work that takes too long

Book a trip, meaning a flight, then a hotel, then a car. Wrap all three in one transaction and you hold locks for the entire booking while a human decides whether they want the aisle seat, and everything touching those rows queues behind you. Garcia-Molina and Salem wrote this up at SIGMOD in 1987: long-lived transactions hold resources for long periods and badly delay the shorter, more common transactions around them.

Their answer was the saga, which replaces one long transaction with a sequence of short ones that each commit immediately, pairing every step with a compensating step that undoes it. If step four fails, you walk backwards through the compensators for three, two and one.

Compensation is not rollback. You cannot recall a confirmation email or erase a charge the way a database undoes a write, so you cancel the booking, issue the refund, send the correction. The undo happens in meaning rather than in bytes, which makes it visible to everyone and sometimes involves apologizing to a real person.

That changes how you order steps. Reversible work goes before the step past which you cannot turn back, which goes before work that can only move forward. Get it wrong and you will need to compensate something that has no compensator.

The cost is isolation, which a saga gives up by construction. Isolation is the promise that nobody sees your work until it is finished, and a saga cannot make that promise because its steps commit one at a time. Other processes will observe your partially completed work, and you decide in advance whether that is acceptable for each step.

Remember the phrase long-lived transaction. A model call taking thirty seconds is one, and a human approval taking three days is an extreme version.

Sagas still assume the other side cooperates. What happens when it won't?

2007 to 2019: giving up gracefully

Two-phase commit needed every participant to speak the protocol, and then the internet arrived and almost nothing did. Neither Stripe nor S3 nor whatever external API you depend on implements XA, and none of them ever will, so for most real operations the bigger transaction is not slow or awkward but unavailable.

Pat Helland had spent much of his career advocating for exactly these guarantees before publishing Life Beyond Distributed Transactions: an Apostate's Opinion in 2007. The title is not ironic. People building large systems had already stopped assuming distributed transactions regardless of the textbooks, and his argument was that the field should work out what to do instead.

What they worked out is a chain of three answers, each one forced by the answer before it.

One. Since you can never know whether your call happened, stop trying to know and make it safe to run twice. That property has a name, idempotence: running something twice leaves the world in the same state as running it once, which is not the same as getting the same answer back twice.

Recognizing a repeat requires a stable identity on every action. That identity is an idempotency key, and Helland wrote the definitive treatment in 2012 as Idempotence Is Not a Medical Condition. With it you stop chasing exactly-once delivery, which is not achievable, and settle for at-least-once delivery plus idempotent processing, which behaves like once.

Two. That handles the external call but not the agreement between it and your own database. Since the two cannot commit atomically, fall back on 1978 and write the intent down first: in one transaction in your own database, the business row and a row saying "I intend to charge this card, under this key." Either both land or neither does, because both sit inside the same guarantee. A separate worker then reads intent rows and performs the calls, retrying until confirmed.

This is the transactional outbox, and the failure it fixes finally gets its name, the dual write problem: two writes to two systems, no coordination, and a window in the middle where a crash leaves you inconsistent. That window is the one in your agent loop. The outbox itself is not new, it is write-ahead logging moved one layer out of the database and into your application.

Three. The last piece arrived when people tired of writing saga logs and outbox relays by hand, badly. Durable execution engines record every completed step to a log that is only ever appended to, then replay it after a crash to rebuild where the process had reached. The lineage runs from Amazon SWF through Cadence, built by Maxim Fateev and team at Uber in 2015 and open sourced in 2017, to Temporal in 2019. Replay only reconstructs the past if running your code again makes the same decisions, which is why determinism stops being a matter of taste in these systems and becomes structural.

That is the inheritance: identity, intent written first, semantic undo, recorded steps. So how much of it did agent frameworks actually pick up?

Where agents sit

Two things write to your database on every turn: the framework's checkpointer, saving loop state, and your own tool code, writing the orders and the charges. In many production systems they point at the same Postgres instance and are never in the same transaction, which makes them the dual write problem arriving as a framework default rather than as anyone's decision.

The consequences are sitting in public issue trackers, three of them describing the same seam:

LangGraph #9006 asks what the intended contract even is when a worker dies in the middle of a tool call. The reporter's framing is the sharpest summary I have read: many traces filed as "the agent failed" are really this seam, where the model never chose badly and the harness duplicated the work.
Strands #4338 reaches it from a different angle. A hook raises after the tool completes, the result message never gets appended, and the conversation is left ending in an unresolved tool call that the agent replays. The issue names payments and order creation explicitly.
Microsoft Agent Framework #7458 executes an approved tool before the run can fail, leaving the retry indistinguishable from a request that was never pending.

So far this is rediscovery, and the microservices answer applies unchanged. Agents add two wrinkles that are in none of the older papers.

The first is that the idempotency key cannot come from the model. Deduplication only works if a repeat is recognizable, and a model that plans again will express the same intention with different wording, different argument order, sometimes a different tool. The second attempt is semantically identical and syntactically new, so a check built on the request body sails past it and the card is charged again. The key has to come from somewhere stable and outside the model's control, which in practice means position in the plan, something like run_id:step_index.

The protocol has not caught up either. MCP, the Model Context Protocol that most agent tooling now speaks, has no idempotency key field, and JSON-RPC request IDs are regenerated on every retry, so a retried tool call is indistinguishable from a new one at the wire level. SEP-3182 proposed fixing that with an idempotencyKey alongside arguments on tools/call, rejecting any key reused with different arguments. It has been closed.

The second is that the determinism requirement is violated by design. Durable execution rebuilds state by replaying your code, assuming it makes the same choices twice, and you have installed the least deterministic component ever shipped at the center of the loop. The way out is to treat the model call as a result to look up rather than logic to execute again, recording the completion when it first happens and reading it back on replay.

Four separate papers in 2026 independently rebuilt transactional machinery for agents, which is decent evidence the gap is real. It is also worth noticing that Cordon's effect outbox is the transactional outbox and SagaLLM's compensation is the 1987 saga, so the field is working this out from scratch rather than reading it.

None of which you need in full to stop the double charge. What is the smallest version?

The smallest thing that fixes it

One table and one rule. The rule is from 1978: write down what you are about to do, before you do it.

create table agent_actions (
  key        text primary key,
  tool       text        not null,
  args       jsonb       not null,
  status     text        not null default 'pending',
  result     jsonb,
  created_at timestamptz not null default now()
);

Every tool call now begins by looking up its key, and there are only three things it can find.

What you find	What it means	What you do
No row	First attempt	Write the row, then call the tool
Row marked `done`	It already happened	Return the stored result, do not call
Row marked `pending`	An earlier attempt died mid-call, and you cannot know whether it landed	Call again, with the same key

That is the entire design. The code is that table:

def run_tool(conn, run_id, step, tool, args, tools):
    key = f"{run_id}:{step}"          # identity from position, never from the model

    with conn.transaction():
        claimed = conn.execute(
            "insert into agent_actions (key, tool, args) values (%s, %s, %s) "
            "on conflict (key) do nothing returning key",
            (key, tool, Json(args)),
        ).fetchone()

    if claimed is None:                                  # a row already exists
        prior = conn.execute(
            "select tool, args, status, result from agent_actions where key = %s", (key,)
        ).fetchone()

        if (prior.tool, prior.args) != (tool, args):
            raise PlanDivergence(key, prior.tool, tool)  # see below

        if prior.status == "done":
            return prior.result                          # row two: already happened

        # row three: still pending, so fall through and call again, keyed

    result = tools[tool](**args, idempotency_key=key)
    conn.execute(
        "update agent_actions set status = 'done', result = %s where key = %s",
        (Json(result), key),
    )
    return result

Why this works: the row commits before the tool is called. A crash after that point always leaves a note behind, so the resumed run finds the row instead of a blank slate and never gets to decide freshly about something already done.

Row three is the honest one. It does not pretend to know whether the charge landed, it hands the same key to the payment provider and lets their deduplication settle it. That is the at-least-once plus idempotency bargain from the last section, in one branch.

There is a fourth thing you can find, and it is the one that catches people out. The row exists, but for a different action, because a key built from position assumes step three means the same thing on every attempt and a replanning model can put something else in that slot. Without the check you would hand an old result to a new intention. Refuse instead. MCP's own idempotency proposal reached the same conclusion when it specified that a key reused with different arguments is rejected outright.

What each additional piece buys

So what does that leave unsolved? Three things, and you will hit them in this order.

One. A row can sit in pending forever if the process that claimed it never returns, so you add a worker that scans for stale rows and finishes them. That worker is the outbox relay, and it also lets your agent step commit the intent and return immediately rather than blocking on a slow external call.

Two. A plan with several steps can still end up half applied, because step three succeeding does not help when step four fails permanently. So each reversible step gets a compensator, the irreversible ones move to the end, and you walk backwards on failure. That is the 1987 saga, and you need it only once an action spans more than one external system.

Three. A long run can lose expensive model calls on every crash, and asking the model again means the replayed agent may take a different path than the one you have already partly executed. So you record model completions alongside tool results. That is durable execution, and by this point adopting Temporal or DBOS is usually cheaper than continuing to build it.

Then break it deliberately, because almost nobody does and it is the only way to know any of this works.

Kill the process after the charge but before the status update. The next attempt should find a pending row and lean on the key rather than charging blindly.
Kill it between the business commit and the checkpoint write, which is the bug from the top of this article. The resumed run should find the action row and skip the call.
Kill it halfway through a plan with several steps. The compensators should run in reverse order.

Each is about five lines of test, and each corresponds to an outage you would otherwise have later.

Memories, guesses and apologies

All of that is machinery. It is worth asking what the machinery is for, and the best answer I know predates every framework in your stack.

In 2009, Helland and David Campbell wrote Building on Quicksand, arguing that once you accept that no component has complete knowledge, the only honest model left is memories, guesses and apologies. You remember what you have seen, you act on incomplete information and are sometimes wrong, and you build a mechanism for making it right afterwards.

That maps onto an agent almost too neatly: the model is the guess machine, the message list is the memory, and the compensating transaction is the apology, which is the only one of the three most agent stacks have not built. We assembled the guess machine, bolted the memory onto the side, and the apology is still missing from most production systems.

The work of figuring out what that apology should look like was done decades ago by people already burned by the same class of failure, and the papers are all still sitting there.

Did your AI agent quietly blow all your money?

Tisha Chawla — Sat, 05 Sep 2026 08:05:32 +0000

Featured at Microsoft Developer - Open to Contributions

You set a per-request limit. Every call passes it, no exceptions.

Then your agent gets stuck in a loop: re-searching, re-checking, retrying the same step over and over. Every single one of those calls is still technically within limits. Nothing was watching the workflow as a whole, only each call, one at a time, blind to a loop right in front of it.

A workflow that should cost $2 comes in at $20 instead. Run that workflow 10,000 times a day and the gap is not $18, it's $180,000.

That blind spot is real, and it's measurable.

TokenOps closes it: one budget for the whole run, checked before every call instead of reported after the fact.

theagentplane / tokenops

Run-aware token governance for multi-agent systems.

TokenOps

Cuts wasted agent spend by up to 65%, governing what the run has already spent before every call.
Toward token governance as a first-class discipline, not an afterthought.

Built by Susheem Koul and Tisha Chawla

See it stop a run mid-budget in the Quickstart below.

Core features · Quickstart · Quickdeploy · How it compares · Policies · Support · Contributing

🙌 Open to contribution

Token spend deserves the same first-class attention as compute or latency, and we are growing the community working on that. Policies, actuators, and the shared ledger are all open to extension. See CONTRIBUTING.md to get started.

✨ Core features

An AI agent's workflow can run up cost fast: dozens of small, individually cheap steps that quietly add up to a surprisingly large bill. TokenOps sets a single budget for the whole workflow and enforces it before every step so spending never gets away…

View on GitHub

🎬 See it happen in ten seconds

pip install agent-tokenops
python -m tokenops.demo

An agent makes 40 model calls. Budget for the whole run: $2.00.

  without TokenOps   40 calls run, spend $5.80
  with TokenOps      halted at call 12, spend $2.03

  $3.77 not spent. The run stopped itself.

No install beyond the package, no API keys, no server. No single call in that run was expensive. It was the 40 of them together that crossed the cap, exactly what a per-request limit can't see.

🔧 Wire it into your own agent without reading the docs

TokenOps ships an integration skill, a written procedure your coding assistant reads and follows so you do not have to.

In Claude Code, from a clone:

/integrate-tokenops

In Cursor, Copilot, or anywhere else, paste this:

Integrate TokenOps into this agent, following https://github.com/theagentplane/tokenops/blob/main/.claude/skills/integrate-tokenops/SKILL.md

It reads your agent's code, picks the right setup, wires the one enforcement point, and tells you what to check. That's the whole integration.

✨ What it does

Feature	What it means
Run-scoped budget	One cap for the whole workflow. Your finance team does not care that each call was "technically within limits."
Enforced pre-call, not "oops, sorry" after	Checks the total before the call goes out and refuses the one that would break budget, instead of quietly logging it for you to discover in an invoice.
Cross-process	Research, summarize, and review can be three separate services sharing one ledger, instead of each one secretly assuming it gets the whole budget to itself.
Steers, not just stops	Shrinks the next prompt, swaps to a cheaper model, or tells your agent it's going in circles, before it pulls the plug entirely.
Tool calls count too	That search result your agent stuffed into the next prompt is spend whether an LLM produced it or not. TokenOps counts it.
Ten policies included	Documented, swappable. Bring your own if ours do not fit.

Not a gateway, not a tracing dashboard. It governs the run, alongside whatever routing and observability you already have.

🤝 Come build it with us

TokenOps is early, 0.x, still finding its shape.

Two areas take contributions without touching the governance core at all:

A new policy (detector + decision) under src/tokenops/control/policies/. Ten existing ones as reference, one doc each in docs/policies/.
A new adapter for another agent SDK, under src/tokenops/adapters/.

Open issues tagged good first issue and help wanted right now, including a dependency pin and a UI responsiveness fix. Small, self-contained, good first PR material.

Get set up:

git clone https://github.com/theagentplane/tokenops.git
cd tokenops
make install
make lint
make test

Open an issue before a bigger PR so we can agree on approach first; typos and small docs fixes can go straight to a PR.

Questions or integration help: Slack · GitHub Discussions · office hours

If this saved you from a bill you didn't see coming, or just made you check your own agent's budget, ⭐ star the repo. It genuinely helps more people find it, and it costs you nothing but a click.

Built by @tisha and @susheem-k

[Boost]

Tisha Chawla — Fri, 04 Sep 2026 06:10:32 +0000

Tisha

Sep 3

Your AI agent can quietly blow 65% past budget

#opensource #python #ai #llm

3 min read

Token spend deserves the same first-class attention as compute or latency, and we're growing the community working on that.

Tisha Chawla — Thu, 03 Sep 2026 23:45:44 +0000

Tisha

Sep 3

Your AI agent can quietly blow 65% past budget

#opensource #python #ai #llm

3 min read

[Boost]

Tisha Chawla — Thu, 03 Sep 2026 21:25:41 +0000

Tisha Chawla

Jul 22

You Recorded the Incident. Now Prove Your Fix Actually Works.

#ai #llm #testing #machinelearning

6 min read

TokenOps video is out now from AI Engineer WF - SF

Tisha Chawla — Sun, 23 Aug 2026 21:14:28 +0000

https://youtu.be/GJX19pNhmSw

Just published TokenOps in Microsoft Command Line! ⭐

Tisha Chawla — Fri, 14 Aug 2026 11:42:52 +0000

If you find it useful, star the repo ⭐: https://github.com/theagentplane/tokenops

Quick takeaways:

Treat tokens like infrastructure, not a surprise bill
Track cost per agent run in real time
Set clear budgets and limits before agents go off track
Useful for anyone building production AI agents Read it here: https://commandline.microsoft.com/tokenops-real-time-run-scoped-cost-control-ai-agents/

Try it:

pip install agent-tokenops

Just published https://commandline.microsoft.com/tokenops-real-time-run-scoped-cost-control-ai-agents/ TokenOps helps devs control AI agent token costs in real time, per run. Star ⭐: https://github.com/theagentplane/tokenops

Tisha Chawla — Fri, 14 Aug 2026 10:14:32 +0000

TokenOps: Real-time, run-scoped cost control for AI agents

Budget caps and gateways can’t stop a runaway AI agent mid-run. TokenOps enforces one shared, run-scoped budget in real time, before the bill arrives.

commandline.microsoft.com

GitHub - theagentplane/tokenops: Run-aware token governance for multi-agent systems. · GitHub

Run-aware token governance for multi-agent systems. - theagentplane/tokenops

github.com

Debugging Multi-Agent Systems: Your Trace Tree Is Lying

Tisha Chawla — Wed, 12 Aug 2026 10:28:32 +0000

Co-written with @susheem-k / @tisha. We build Chronicle in the open at AgentPlane. Originally published at theagentplane.github.io.

It's Tuesday. Someone on support forwards you a message: "priya@acmecorp.com says the assistant told her something false, sometime yesterday afternoon." No conversation id. No message id. Just a name and a rough window of time. Maybe it's quieter than that: a thumbs-down on your own feedback button, or an alert your monitoring auto-files. Either way, what lands on your desk is never "here is the broken function." At best, it's a person and roughly when.

Your agent, meanwhile, is not one function call. A single user message might fan out into an orchestrator calling a researcher, which calls a model, which calls a search tool, twice, with a retry in the middle. That one bad reply the user saw was produced by one specific call, three levels deep, somewhere inside a tree of a dozen calls made sometime in a two-hour window. You have a name and a time range. You need a call stack.

This post closes that gap, from first principles, with the actual mechanism, not a hand-wave. By the end you'll know how to go from "a name and roughly when" down to a session, a message, and the exact call that misfired; the specific attribution bug that quietly breaks naive tracing in any system with parallel or repeated sub-agents (you have this bug right now if you haven't specifically fixed it); and the one-line change that fixes it, drawn from Chronicle (chronicle#41, open for review as we write this).

First principles: the four IDs

Before you can go from "a name and roughly when" to "here's the exact call that misfired," you need to know what identifies what, and where you actually start. There are four levels, and conflating any two of them is where most homegrown tracing setups go wrong.

Session (session_id). One conversation. A user might send you ten messages over an hour; they all share one session.
Message. One user turn inside that session. This is what a precise bug report gives you, when you're lucky enough to get one: "in this message, the agent said something wrong."
Trace (trace_id). One agent run. In practice, one message turn produces one trace: the user sends a message, the agent does whatever it does (one call or twenty), and produces a reply.
Envelope (envelope_id, OTel calls it a span). One LLM call, one tool call, or one routing decision inside that run. A trace is an ordered tree of these.

Resolving "yesterday afternoon" into a session_id is ordinary application code, not Chronicle's job: look up the account, find the sessions active in that window. Chronicle's job starts once you have that session_id. Single-shot service-to-service calls collapse the next two (no multi-turn session, just a session_id and maybe a caller_id). Multi-turn chat keeps session_id constant across the conversation and mints a new message_id (and a new trace_id) on every turn. Either way, the shape is the same: you start with almost nothing, resolve down to a session and a message, and Chronicle gets you the rest of the way to the exact call.

This is exactly the mapping problem we'd sketched out on a whiteboard before writing a line of code: name and time → session → message → trace, with the open question of how you ever get from "roughly yesterday afternoon" to "the actual execution graph." The rest of this post is the answer.

Why "just add tracing" doesn't get you there

Say you've done the obvious thing: wrapped your LLM and tool calls so each one records an envelope with a parent_envelope_id, and you build a tree out of that after the fact. This works in every demo. It works in your first three fixtures. Then it breaks silently, on the one trace you actually needed, because you have two branches of the same shape running in the same trace, which in a multi-agent system is the common case, not the edge case.

And if you already run OpenTelemetry against this agent, you are not exempt. The failure mode here isn't "we have no tracing." It's "our tracing is confidently wrong," which is worse, because a tree that renders cleanly is a tree you trust. You stop looking anywhere else. You spend the debugging session inside the wrong subtree, close the ticket with a fix that doesn't touch the actual bug, and it comes back a week later with a slightly different repro.

Here's the concrete scenario: an orchestrator calls the same sub-agent twice, once per source it needs to research. Each call does an LLM planning step and a tool call:

from chronicle import boundary

@boundary("planner", kind="llm")
def planner(task: str) -> dict:
    ...

@boundary("web_search", kind="tool")
def web_search(query: str) -> dict:
    ...

def researcher(task: str) -> dict:      # plain code, not a boundary
    plan = planner(task)
    return web_search(plan["query"])

def orchestrator(task: str) -> dict:    # plain code, not a boundary
    first = researcher(task + " (source A)")
    second = researcher(task + " (source B)")
    return [first, second]

Notice researcher and orchestrator aren't boundaries themselves. That's deliberate: you mark decision nodes (an LLM call, a tool call, a routing choice), not the whole call stack. This keeps recording overhead near zero and keeps your orchestration as plain code Chronicle never has to know about. The question is how those decision nodes learn who their parent is if nobody wrapped researcher or orchestrator.

The naive answer is "attribute a new envelope to whichever envelope finished most recently." It's the simplest thing that could work, and it's wrong the moment two sibling subtrees are in flight or interleaved:

When researcher's second call starts its planner call, the "most recently finished" envelope is web_search#1 from the first call, not anything belonging to the second researcher. The tree you reconstruct silently welds the second research branch onto the first one. You go looking for why researcher #2 produced a bad answer, and every log line in front of you belongs to researcher #1. Nothing crashes. Nothing throws. The trace just quietly points you at the wrong code for an hour.

The fix: open the span before the body runs

The fix is one sentence: a boundary opens its span when it's entered, before the wrapped function runs, not after it returns. That single change is the difference between "attribute by whatever happened to finish last" and "attribute by what's actually on the call stack right now," and that is precisely what OpenTelemetry's Context does, and precisely what was missing.

# Simplified: what @boundary now does around your function
span_id, parent_id = session.start_span()   # push onto the active-span stack
try:
    result = fn(*args, **kwargs)            # anything this calls sees `span_id` as parent
finally:
    session.end_span()                      # pop back to the caller's span

Because start_span() runs before fn, any boundary invoked while this one is still on the stack correctly parents to it, no matter what else happens to finish in between. Rerun the two-branch scenario above and the tree comes out right every time, regardless of timing:

Trace: trace-9f2a1
  total: 164.5ms   resource/dims: message_id=msg_042 session_id=sess_abc

orchestrator#1   ████████████████████████████████████████████  164.5ms  custom
  researcher#1   ██████████████████████                          74.8ms  custom
    llm#1         ██████████████                                  42.3ms  llm
    web_search#1                ██████████                        31.7ms  tool
  researcher#2                          ████████████████████      72.3ms  custom
    llm#2                                ████████████              40.5ms  llm
    web_search#2                                       ██████      30.9ms  tool

This is what graph.to_otel_waterfall() prints (there's also graph.to_otel_tree(), which prints the same tree as indented lines instead of a timeline). Both are debug helpers you run locally or pipe into a log line, not a hosted product; if you want a shared dashboard your whole team browses, that's a separate concern from what a recording library should own.

The other half: getting `session_id` and `message_id` onto the trace

Correct nesting gets you a trustworthy tree once you're looking at the right trace. It doesn't yet tell you which trace, out of everything your agent ran today. For that, Chronicle added dims: a flat dict[str, str] you pass once, that gets copied onto every envelope in the run.

import chronicle

with chronicle.record(
    "trace-9f2a1",
    store=".chronicle/runs/prod.jsonl",
    dims={
        "session_id": "sess_abc",   # constant for the whole conversation
        "message_id": "msg_042",    # new value every turn
    },
):
    orchestrator(user_message)

That's it. Chronicle doesn't store your chat history and doesn't own a "look up trace by message ID" index; that's a product concern for whatever's on the other end (your logging pipeline, your control plane, a support tool). We're building a complementing dashboard for exactly this lookup next (more on that in a future post). Until then, what Chronicle guarantees is that once you have a session_id and a message_id, every envelope in the matching trace carries them, so any store you point it at can build that index trivially: grep, a SQL WHERE, or a dashboard query, your choice.

Try it yourself

pip install agent-chronicle

Mark your decision nodes. Wrap the LLM calls, tool calls, and routing choices you'd actually want to assert on in a test, not the whole orchestrator function.

   from chronicle import boundary

   @boundary("planner", kind="llm")
   def planner(task: str) -> dict: ...

   @boundary("web_search", kind="tool")
   def web_search(query: str) -> dict: ...

Record a run with the ids you'll get back from a ticket.

   import chronicle

   with chronicle.record(
       "trace-9f2a1",
       store=".chronicle/runs/prod.jsonl",
       dims={"session_id": "sess_abc", "message_id": "msg_042"},
   ) as session:
       orchestrator(user_message)

When a ticket comes in, pull the trace and print the tree.

   graph = chronicle.ExecutionGraph.from_envelopes(session.trace_id, session.envelopes)
   print(graph.to_otel_waterfall())

Turn the bad run into a regression test. Add export="fixtures/traces/incident-001/" to the record(...) call and the exact trace gets committed to git as a fixture. Replay it (chronicle.replay_trace) with your fix applied and no live model calls, so "did this actually fix it" becomes a test you run in CI, not a hope.

If your agent is instrumented with OpenTelemetry already, chronicle.instrument_otel() emits the same nested spans to your existing collector (pip install agent-chronicle[otel]), so this isn't an either/or with the observability stack you already run.

What this doesn't solve, on purpose

We'd rather tell you the edges than let you find them the hard way:

Chronicle doesn't store your session or message history. It stamps the ids you give it onto envelopes. Owning "what messages exist in this session" is your app's job, not a recording library's.
There's no hosted lookup UI yet. to_otel_tree() / to_otel_waterfall() are local debug output. A dashboard where a support ticket resolves straight to a trace view is a control-plane concern we're building next, not something bolted into this library.
Replay proves control flow, not answer quality. Replaying a trace with your fix tells you the agent takes the right path and calls the right tools with the right arguments, deterministically, with no live model calls. It won't tell you a subjectively "better" answer is actually better; that's a model-quality eval, a different tool for a different question.

If this is a problem you have

Star theagentplane/chronicle 🌟 if this is useful, it's the fastest way to tell us to keep going, and it's genuinely how a two-person OSS project gets found by the next person with this exact problem. Try it on one flaky agent, tell us where the API gets in your way, or open an issue with the trace that broke you. We're building this in public specifically so the next fix is shaped by a real incident, not a guess.

We're @susheem-k and @tisha, building agent infrastructure in the open at AgentPlane. Chronicle is one piece; TokenOps (run-aware token governance) is the other. Follow along or browse the org.

theagentplane / chronicle

Record-and-replay for agent decision graphs: reproduce a prod agent failure as a committed regression test, and re-run your fix without live LLM calls.

Chronicle

Record-and-replay for agent decision graphs.
Turn a production agent failure into a committed regression test, and re-run your fix without live LLM calls.

Watch the full walkthrough

Chronicle records what your agent did at each decision point (its LLM calls, tool calls, and routing choices) so you can reproduce a production failure as a committed regression test and re-run your fix without live LLM calls. It targets one specific real problem: control-flow and tool-safety regressions in multi-agent systems, caught deterministically from recorded incidents.

New here? The step-by-step onboarding guide walks you from install to a committed regression test.

Why · Quick start · Cut-point replay · Recording · Verification · Compare · Demos · FAQ · Roadmap

Key terms (boundary, Envelope, trace, dims, fixture, stub, live, cut-point)

Term	What it means
Boundary	A decision node you mark: an LLM call, a tool call, or a routing choice — not

…

View on GitHub

40% of AI Agent Projects Will Be Canceled by 2027. Token Governance Is How You Avoid It.

Tisha Chawla — Sun, 09 Aug 2026 10:55:47 +0000

By Tisha Chawla and Susheem Koul

TL;DR. AI agents spend tokens to think, and how many they spend is nondeterministic. The same task can cost wildly different amounts across runs, and a stuck recursive loop can spend without limit while still looking successful. You cannot budget a nondeterministic resource up front. You have to observe it, attribute it, and enforce limits at runtime, from a control loop kept separate from the agent.

🛠️ We built this into an open-source tool. TokenOps is run-aware token governance for multi-agent systems: a control plane + SDK that caps spend and steers behavior across a whole agent workflow, not per request.
pip install agent-tokenops
If this field guide is useful, ⭐ star the repo so more people building agents can find it.

First principles
The problem
Primitives for token governance
The current space
The discipline
Where TokenOps fits

0. First principles

Start from what is provably true, then build up.

A token is a metered unit of a paid resource. Every model call consumes a countable number of tokens at a published price.¹²
Consumption is nondeterministic. Token generation is stochastic: at each step the model samples the next token from a probability distribution, so the same prompt can return a different completion, and a different number of output tokens, on every run. OpenAI states its APIs are nondeterministic by default, and that determinism is not guaranteed even with a fixed seed.³ Agents amplify this. An agent picks its next action from that nondeterministic output, so the number of reasoning steps and tool calls, and therefore the total token spend, is a random variable, not a fixed cost.
You can only govern what you can measure, attribute, and bound. Measure it (how many), attribute it (whose), bound it (a ceiling that actually holds).
You cannot bound it by prediction. A static estimate set in advance is wrong on the tail. The only bound that holds is enforced at runtime: watch spend as it accrues, and stop the run when it crosses a limit.
The bound must sit outside the agent. An agent chooses its own next call, so nothing in its own logic is guaranteed to stop it. Governance is an external control loop around the agent, not a feature inside it.

The three questions that decide whether you govern token spend:

How many tokens did this run spend? (measure)

Who spent them: which user, which agent, which run? (attribute)

What stops a run before the invoice does? (bound)

If you cannot answer all three, you are observing spend after the fact, not governing it.

1. The problem

One example, used throughout. Agent A is a research agent: it calls a search tool, reads the results, and hands findings to Agent B, a summarizer. Agent A runs on OpenAI, Agent B on Anthropic. We follow this single task through every failure mode below.

What a token is. A token is a chunk of text mapped to an integer ID from the model's fixed vocabulary. A subword tokenizer (for example, byte-pair encoding) splits text into these pieces, so one token can be a whole word, part of a word, or a single character.¹⁴ Different models use different tokenizers, so the same sentence is a different token count on each. Rule of thumb in English: roughly 4 characters, or 0.75 words, per token.¹ Output tokens cost several times more than input tokens.⁵²

Why tokens became an ops problem. Agents do not make one call, they loop. Anthropic measured agents using about 4x the tokens of a chat, and multi-agent systems about 15x, with token usage alone explaining roughly 80% of the variance in cost.⁶ In our example, Agent A might call search once, or twenty times. You do not know which until it runs.

1.1 The token leak

Agent A calls search("pricing"). The result is weak, so it feeds its own output back in and calls search("pricing") again, and again. This is a recursive loop: the agent keeps invoking the same step because nothing tells it to stop. Fifty calls later it still returns a clean summary, so every functional dashboard shows success. The spend shows up in exactly one place: the bill.

That is a token leak: spend that grows with no bound and no attribution, whether from a silent retry loop, a context window that keeps growing, or an unflagged model upgrade. Provider guardrails do not catch it: OpenAI project budgets are alerts, not hard caps.⁷ Unmanaged, this is now a top failure mode. Gartner predicts over 40% of agentic AI projects will be canceled by end of 2027, citing escalating cost.⁸

The quieter leak: cache busting. Repeated input can be cached at a steep discount: OpenAI documents 50% off cached input,⁵ and Anthropic charges roughly 90% below base input for cache reads.² The catch: caching matches the prompt from the top down, only up to the first point where the text differs. If Agent A pastes a current_timestamp or session_id near the top of its system prompt, it throws away the cache for everything below it, and a large context that should cost a tenth of full price pays full price every turn. Same text, same step count, 10x the bill. The fix is structural: put static content (system prompt, tool definitions) first and volatile values last, and treat a drop in cache hit rate as a spend alarm on its own.⁵

A third leak: reasoning bloat. Reasoning models (OpenAI's o-series and GPT-5 reasoning, Anthropic extended thinking) spend hidden reasoning tokens server-side before emitting a single visible token, and those tokens are billed.⁹ Two consequences. First, max_output_tokens caps the sum of reasoning and visible output,⁹ so a runaway reasoning loop can burn the entire budget inside one call without returning a character. Second, if your breaker trips by parsing the output stream, it is blind during the thinking phase: the spend has already happened before the first chunk arrives. Read the usage totals the model reports, not just the visible stream.

1.2 Distributed token lineage

The monthly bill arrives: one total from OpenAI, one from Anthropic. Neither number tells you that Agent A's recursive loop caused most of the cost, or which user started the run. Lineage is the missing label on every call: which user, which agent (A or B), and which run produced it. OpenTelemetry's GenAI conventions define exactly these attributes: gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, plus agent and session identifiers.¹⁰ Recent versions also add attributes for the costly hidden categories, cached and reasoning tokens, mirroring the provider payloads: OpenAI's prompt_tokens_details.cached_tokens and completion_tokens_details.reasoning_tokens, and Anthropic's cache_read_input_tokens.¹⁰ Track those, or your lineage misses exactly the tokens most likely to surprise you.

One more catch: Agent A's OpenAI tokens and Agent B's Anthropic tokens are not the same unit, because the tokenizers and prices differ. You cannot add raw tokens across providers. Convert each call to cost first (one denominator, such as micro-dollars), then sum and enforce. Enforce on cost, not tokens.

1.3 Isolate the guard

Put the budget check inside Agent A's loop and two things break: the same bug that makes it loop can skip the check, and Agent B needs its own copy. So keep two planes apart:

Data plane: the agent doing the work.
Control plane: the code that meters, attributes, and halts spend.

The meter and the limit live in the control plane, an out-of-band loop both agents call but neither can edit or bypass.

2. Primitives for token governance

Six building blocks. Each answers one question. Together they are the vocabulary for everything below.

Primitive	What it is	Question it answers	Source
Meter	Counts tokens and cost on every call	How much did this consume?	¹¹
Attribution context	Tags every call with user, agent, run	Whose spend is this?	¹⁰
Budget / quota	A hard ceiling on tokens or cost over a window	What is the cap, and does it hold?	¹²¹³
Rate limit	A cap on tokens or requests per unit time	How fast is too fast?	¹⁴
Circuit breaker	Trips and fails fast when a condition crosses a threshold	What stops a bad run immediately?	¹⁵
Control plane	Where the rules live, out of band from the agent	Who owns the decision?	§1.3

The circuit breaker, precisely. You wrap a protected call in an object that monitors it. When a chosen condition crosses a threshold, the breaker trips to an open state and fails fast, returning an error without making the call. Classic breakers add a half-open state to probe recovery.¹⁵ A failure can be any condition you define, such as a timeout or an HTTP 429. For token governance, the condition is behavioral: a repeated action, a spend velocity, or a collapse in cache hit rate. The breaker trips to halt the run before the spend lands.

How a behavioral check actually works. You do not need brittle, domain-specific rules. Three system-level signals over an agent's step history cover most runaway modes:

Semantic loop. Hash each tool call's name and arguments, and trip when the same signature repeats inside a sliding window. A vector-similarity check on consecutive prompts catches near-duplicates, such as an agent feeding the same error back to a tool.
Spend velocity. Track cost per step (the derivative of cost over steps). A sharp spike means the context is compounding, for example appending full raw logs on every failed turn.
Efficiency decay. Watch the ratio of progress made to tokens burned. This one is the most heuristic and hardest to define cleanly, so treat it as a warning, not a hard trip.

The first two are cheap and robust. Start there.

2.1 Beyond halt: two richer responses

Halting is the simplest response to a trip, not the only one. Two production patterns are worth knowing, each with a caveat.

Model cascade (graceful degradation). At a warning threshold (say 75% of a session budget), the control plane swaps the next call to a cheaper model and trims non-essential tools, trying a low-cost completion before any hard halt. This ships today: Cloudflare can switch to a cheaper model once a budget is exhausted,¹³ and gateways like LiteLLM support model fallbacks.¹² Caveat: a weaker model can reason worse and loop more, trading a hard failure for a quality risk and sometimes more spend. It is a tradeoff, not a free win.
Escrow and human-in-the-loop. Instead of killing the run, the control plane suspends it, snapshots its state, and pings a human to inspect and resume. Real, but not free: pause-and-resume requires a durable, checkpointable runtime. LangGraph, for example, requires a checkpointer to persist state across an interrupt.¹⁶ A lightweight in-process wrapper cannot serialize and resume an arbitrary call stack on its own.

3. The current space

3.1 Five dimensions

Score every tool on five questions. The first three are table stakes. The last two are where the gap is.

Dimension	Primitive	Question	Source
Observe	Meter	Can you see tokens per call?	¹⁰¹¹
Attribute	Attribution context	Whose spend: user, agent, run?	¹⁷
Enforce	Budget, rate limit	Can you stop spend in real time?	¹²¹³
Behavioral	Circuit breaker	Can you halt on a runaway pattern, not just a dollar line?	⁶¹⁸
Ownership	Control plane	Which layer holds the control point?	§1.3

3.2 Five categories

Every tool sits at one of five layers in the stack.

Layer	What it is	Examples	Stops spend?
Provider	Vendor-native limits	OpenAI, Anthropic, Azure OpenAI	Coarse, often alerts
Gateway	Out-of-process proxy you route through	LiteLLM, Portkey, Cloudflare, Kong	Yes, threshold-based
Component	In-process library inside your app	OpenLLMetry, LangChain callbacks	Rare
Backend	Collector or dashboard	Langfuse, Datadog, Arize, LangSmith	No, after the fact
Standard	The wire format everyone emits	OpenTelemetry GenAI conventions	n/a

3.3 The gap

Across these tools, most observe, several enforce a dollar or rate threshold (the gateway layer is moving fast: Cloudflare shipped per-user spend limits in June 2026¹³), but almost none halt a run on pathological behavior (a runaway loop) before the threshold is hit.

Enforcement also tends to live at the gateway, which sees a stream of independent requests, not the in-process context (the agent's step sequence and loop structure) needed to recognize a loop early. Behavioral signals like a collapsing cache hit rate or a semantic loop are easiest to read right next to the prompt structure and steps that caused them, which a gateway does not retain.

In-process, behavioral enforcement is the least served square in the landscape today.

4. The discipline

You already do this for CPU and memory. You measure it, label it, watch it, and cap it. Tokens are simply the newest resource to earn the same four steps. The order matters, because each step depends on the one before it: you cannot attribute what you never measured, and you cannot enforce what you cannot see.

Step	In plain terms	What it does	Mechanism
1. Instrument	Meter it	Emit one record per model and tool call, carrying token counts	OpenTelemetry GenAI spans ¹⁰
2. Attribute	Tag it	Stamp every record with user, agent, and run	OTel attributes and vendor tags ¹⁷
3. Surface	Watch it	Stream the records so a climbing cost or a falling cache hit rate is visible live, not at month end	Observability backends ¹⁷
4. Enforce	Cap it	Apply a budget, a rate limit, or a behavioral circuit breaker that trips on a recursive loop	Budgets and breakers ¹²¹⁵

Walk Agent A and Agent B through it:

Instrument. Every search call and every model call, on both OpenAI and Anthropic, emits a record with its token usage.
Attribute. Each record carries agent=A or agent=B and the run id, so the bill splits cleanly instead of arriving as one number.
Surface. Agent A's recursive search loop appears as a steeply climbing cost line the moment it starts, not four weeks later.
Enforce. When Agent A repeats the same search call past a set threshold, the breaker trips and halts the run before the spend lands.

All four steps live in the control plane (see 1.3), outside the agents, so neither Agent A nor Agent B can skip them.

The four verbs are a way to organize the work, not an official standard. Every mechanism they point to (spans, attributes, budgets, breakers) is real and documented in the footnotes.

5. Where TokenOps fits

The least-served square in section 3.3, in-process, behavioral enforcement that governs the whole run, is exactly the square we built TokenOps to fill.

TokenOps is a control plane + SDK for agent stacks. Entry agents register a run; every LLM and tool crossing shares one run_id and one ledger. Mapping it back to the primitives from section 2:

Govern the run, not the request. One run_id spans every model, tool, and agent-to-agent hop, so the Agent A → Agent B workflow stays inside a single budget, the multi-agent case a gateway can't see.
Shared ledger across processes. Spend, inflight, and halt state live in SQLite, so Agent A and Agent B can't each burn the full cap locally. This is the cross-provider, enforce-on-cost point from section 1.2.
In-path enforcement. wrap_complete runs detect → decide → apply before the next LLM call, the control loop from section 1.3, out of band from the agent.
Steer or stop. Actuators go beyond halt: HALT · MUTATE · INJECT · reject/queue, the richer responses from section 2.1.

pip install agent-tokenops

Its companion, Chronicle, covers the record-and-replay half: it captures decision boundaries so a production incident becomes a deterministic regression test.

TokenOps is early (0.x) and MIT-licensed. If this field guide gave you the vocabulary you were missing, the repo is where it becomes runnable:

👉 github.com/theagentplane/tokenops. If it's useful, ⭐ star it so more people building agents can find it. Issues and PRs welcome.

Built by Tisha Chawla and Susheem Koul.

OpenAI, "What are tokens and how to count them." https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them ↩
Anthropic, "Pricing" (base input vs output; cache-read pricing). https://platform.claude.com/docs/en/about-claude/pricing ↩
OpenAI, "Reproducible outputs with the seed parameter" (APIs are nondeterministic by default; determinism not guaranteed). https://developers.openai.com/cookbook/examples/reproducible_outputs_with_the_seed_parameter ↩
OpenAI, "tiktoken" (open-source tokenizer). https://github.com/openai/tiktoken ↩
OpenAI, "Prompt Caching in the API" (input vs output pricing; cached-input discount). https://openai.com/index/api-prompt-caching/ ↩
Anthropic, "How we built our multi-agent research system" (4x and 15x token multipliers; ~80% of cost variance). https://www.anthropic.com/engineering/multi-agent-research-system ↩
OpenAI, "Managing projects in the API platform" (project budgets are alerts, not hard caps). https://help.openai.com/en/articles/9186755-managing-projects-in-the-api-platform ↩
Gartner, "Over 40% of Agentic AI Projects Will Be Canceled by End of 2027" (June 25, 2025). https://www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-predicts-over-40-percent-of-agentic-ai-projects-will-be-canceled-by-end-of-2027 ↩
OpenAI, "Reasoning models" (internal reasoning tokens are billed; max_output_tokens caps reasoning plus visible output). https://developers.openai.com/api/docs/guides/reasoning ↩
OpenTelemetry, "Semantic conventions for generative AI" (status: Development). https://opentelemetry.io/docs/specs/semconv/gen-ai/ ↩
OpenTelemetry, "Semantic conventions for generative AI metrics" (gen_ai.client.token.usage). https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-metrics/ ↩
LiteLLM, "Budgets and Rate Limits." https://docs.litellm.ai/docs/proxy/users ↩
Cloudflare, "AI Gateway spend limits." https://blog.cloudflare.com/ai-gateway-spend-limits/ ↩
Kong, "Token Rate-Limiting and Tiered Access for AI Usage." https://konghq.com/blog/engineering/token-rate-limiting-and-tiered-access-for-ai-usage ↩
Martin Fowler, "CircuitBreaker" (popularizing Michael Nygard, Release It!). https://martinfowler.com/bliki/CircuitBreaker.html ↩
LangChain, "Persistence" (LangGraph checkpointers; pause and resume require a checkpointer). https://docs.langchain.com/oss/python/langgraph/persistence ↩
Helicone, "Cost Tracking." https://docs.helicone.ai/guides/cookbooks/cost-tracking ↩
Portkey, "Budget Limits." https://docs.portkey.ai/docs/product/ai-gateway/virtual-keys/budget-limits ↩

You Recorded the Incident. Now Prove Your Fix Actually Works.

Tisha Chawla — Wed, 22 Jul 2026 16:50:02 +0000

Part 2 of Your Agent Failed in Prod. Good Luck Reproducing It.

This work was presented at the AI Engineer World's Fair 2026 by Susheem Koul and Tisha Chawla.

In Part 1 the agent deleted the wrong record at 9:04, and you could not make it happen again. The fix was not to force the model to be deterministic. It was to record the run: the exact prompt, the sampled completion, the tool calls, the retrieved chunks, the pinned model version. Freeze the run, not the model.

So you did that. The incident is now a file on disk. You can open it and see, precisely, what the agent decided that night.

Now comes the part nobody talks about. You still have to fix the bug. And you have to prove the fix works. This is where most teams quietly fall back into the swamp they just climbed out of.

The trap, one more time

You change the code. Now you want to confirm it is fixed. So you run the agent again and watch.

Stop. You just walked back into the trap.

The moment you re-run to check your fix, you are regenerating. The model samples a fresh path. The retrieval returns slightly different chunks. The batch shape on the endpoint is different from the batch shape at 9:04. Your "it works now" is one draw from a distribution, and so was the failure. You have proven nothing, and you know it, because the failure never reproduced on demand in the first place.

To verify a fix you need the exact opposite of a re-run. You need everything about that incident to stay frozen, except the one thing you changed.

A scalpel, not a re-run

That is the whole idea, and it has a name: cut-point replay.

Take the recorded incident. Replay it. But mark one boundary as the cut-point. Everything upstream of it is served from the recording, byte for byte the same inputs the agent saw that night. The boundary you changed runs your new code, live. Everything downstream runs live too, so you can watch what your fix does to the rest of the run.

You are not re-running the agent. You are dropping your new code into the middle of a frozen incident and asking one question: given exactly what happened up to this point, does my change do the right thing now?

No model call. No API cost. No flakiness. The same incident, every time, forever.

Walking through it

Here is the deletion agent from Part 1. Two boundaries: the model decides, the tool acts.

from chronicle import boundary, reset_session, ReplayPlan
from chronicle.envelope.store import EnvelopeStore

@boundary("agent", kind="llm")
def agent(state: dict) -> dict:
    ...  # calls the model, returns a decision and tool calls

@boundary("delete_file", kind="tool")
def delete_file(path: str, environment: str) -> dict:
    ...  # the ungated tool that wiped prod at 9:04

You already recorded the incident and froze it as a fixture:

session = reset_session()
session.store = EnvelopeStore(".chronicle/runs/incident.jsonl")
session.begin_trace("deletion-incident-001")
run_agent(...)                                    # the bad run, captured
session.export_trace("fixtures/traces/deletion-incident-001/")

The recorded graph is exactly what you would expect:

agent@1  ->  delete_file@1 (deleted prod)  ->  agent@2

Now the fix. One guard, in the tool:

@boundary("delete_file", kind="tool")
def delete_file(path: str, environment: str) -> dict:
    if environment == "production":
        return {"blocked": True, "reason": "guard: destructive op refused in prod"}
    ...

And here is the test that proves it, against the real incident:

session = reset_session()
session.load_trace("fixtures/traces/deletion-incident-001/")
session.enable_replay(
    ReplayPlan()
    .stub("agent", 1)         # upstream: the exact decision the model made at 9:04
    .live("delete_file", 1)   # cut-point: your new gated code runs for real
    .live("agent", 2)         # downstream: watch what the agent does after the block
)
run_agent(...)

assert session.captured_result("delete_file", 1)["blocked"] is True

Read the plan again, because it is the entire point. agent@1 is stubbed: the model does not run, you replay the decision it actually made that night. delete_file@1 is live: your new guard executes against those exact arguments. agent@2 is live: you get to see how the agent reacts to a refused deletion instead of a successful one.

You changed one boundary and held the rest of history still. If the guard blocks the delete, the fix works. Not "worked once." Works, deterministically, on the recorded incident, in CI, with no API key.

The two layers, put to work

Part 1 argued for testing in two layers. Cut-point replay is Layer 1 in action: structural, deterministic, about control flow and tool safety. Did the right tool get called? Were the arguments shaped correctly? Was the destructive action refused? None of that needs the model, so none of it flakes.

Layer 2 is for the questions structure cannot answer. If your fix was a prompt rewrite or a model bump, "is the output still correct" is a judgment, not an equality check. That is where an LLM-as-judge scores the new completion against the recorded gold one for meaning, not for bytes. Use Layer 1 to prove the machinery is right. Use Layer 2 to prove the words are still good.

The incident becomes a regression test

Here is the quiet payoff. That fixture under fixtures/traces/ is committed to git. It is now a permanent test. Six months from now, when someone refactors the tool router and the guard silently stops firing, this test goes red on their pull request, not on a customer's production database.

The failure that was unreproducible at 9:04 becomes a green check that runs on every commit. That is the difference between an incident and a regression test: one is a story you tell, the other is a thing your CI enforces.

Before you commit anything: redact

A recorded run is a faithful copy of production. It contains the assembled prompt, the retrieved chunks, the tool arguments. That means it can contain customer names, emails, API keys, internal URLs, whatever your agent touched.

You cannot commit that to git raw. Security and legal are right to block it, and a leaked secret in a fixture is a real incident of its own.

So redaction is not a nice-to-have on the recording path, it is a gate. Scrub secrets and PII out of the envelope before it is written, keep the shape and the structure that your tests assert on, and drop the sensitive values. A recording you cannot safely commit is a recording you will not use.

What this does not fix

Be honest with yourself about the boundary of the technique.

Cut-point replay fixes bugs in your code: routing, guards, argument assembly, tool safety, orchestration. It reproduces those perfectly and lets you verify a fix cheaply.

It does not fix a bad generation. If the model hallucinated a refund amount, replay will faithfully serve that hallucination back to you. Fixing that lives in Layer 2 and in prompt and model work, not in deterministic replay.

Fixtures also drift. Treat them like snapshot tests: when the prompt or the schema changes on purpose, the fixture has to be re-recorded on purpose. And hosted model drift is still outside your control, which is exactly why you pin the version in the envelope so you at least know when it moved.

TLDR;

Record at the boundary, the full run, not just the prompt. (Part 1.)
Reproduce the incident by replaying the recording, no model call.
Pick the boundary you changed. That is your cut-point.
Stub everything upstream from the recording. Run the cut-point live. Watch downstream.
Assert on the cut-point result, the blocked flag, the tool call, the argument, not on the final prose.
Commit the trace as a regression test so the incident can never quietly return.
Redact before you commit. Always.

If you want the tooling instead of building it yourself, our team has put the record, cut-point replay, and two-layer verification into an open-source library called Chronicle: github.com/theagentplane/chronicle. It is early and honest about its limits. Issues and war stories welcome.

Official PyPi Release Here

Install to record and replay your agentic workflow

pip install agent-chronicle

AIE WF Submission

Tisha Chawla — Thu, 09 Jul 2026 07:18:14 +0000

DEV Community: Tisha Chawla

Your Agent Has a 1987 Database Problem

How an agent loop actually works

What happened in the trace

1978: all or nothing

1978 to 1991: trying to leave the box

1987: work that takes too long

2007 to 2019: giving up gracefully

Where agents sit

The smallest thing that fixes it

What each additional piece buys

Memories, guesses and apologies

Further reading

Did your AI agent quietly blow all your money?

Featured at Microsoft Developer - Open to Contributions

theagentplane / tokenops

Run-aware token governance for multi-agent systems.

TokenOps

🙌 Open to contribution

✨ Core features

🎬 See it happen in ten seconds

🔧 Wire it into your own agent without reading the docs

✨ What it does

🤝 Come build it with us

[Boost]

Your AI agent can quietly blow 65% past budget

Token spend deserves the same first-class attention as compute or latency, and we're growing the community working on that.

Your AI agent can quietly blow 65% past budget

[Boost]

You Recorded the Incident. Now Prove Your Fix Actually Works.

TokenOps video is out now from AI Engineer WF - SF

Just published TokenOps in Microsoft Command Line! ⭐

Just published https://commandline.microsoft.com/tokenops-real-time-run-scoped-cost-control-ai-agents/ TokenOps helps devs control AI agent token costs in real time, per run. Star ⭐: https://github.com/theagentplane/tokenops

TokenOps: Real-time, run-scoped cost control for AI agents

GitHub - theagentplane/tokenops: Run-aware token governance for multi-agent systems. · GitHub

Debugging Multi-Agent Systems: Your Trace Tree Is Lying

First principles: the four IDs

Why "just add tracing" doesn't get you there

The fix: open the span before the body runs

The other half: getting session_id and message_id onto the trace

Try it yourself

What this doesn't solve, on purpose

If this is a problem you have

theagentplane / chronicle

Record-and-replay for agent decision graphs: reproduce a prod agent failure as a committed regression test, and re-run your fix without live LLM calls.

Chronicle

40% of AI Agent Projects Will Be Canceled by 2027. Token Governance Is How You Avoid It.

Contents

0. First principles

1. The problem

1.1 The token leak

1.2 Distributed token lineage

1.3 Isolate the guard

2. Primitives for token governance

2.1 Beyond halt: two richer responses

3. The current space

3.1 Five dimensions

3.2 Five categories

3.3 The gap

4. The discipline

5. Where TokenOps fits

You Recorded the Incident. Now Prove Your Fix Actually Works.

The trap, one more time

A scalpel, not a re-run

Walking through it

The two layers, put to work

The incident becomes a regression test

Before you commit anything: redact

What this does not fix

TLDR;

AIE WF Submission

The other half: getting `session_id` and `message_id` onto the trace