DEV Community: Austin Vance

Agentic Software Engineering Makes Human Context the Budget

Austin Vance — Tue, 04 Aug 2026 17:37:06 +0000

Agentic software engineering is moving the bill into human context.

Code is coming faster, pull requests are piling up faster and validation work lands in ever larger piles of judgment work that still need to be handled by people.

This feeling has been captured by Honeycomb’s Douglas Soo in his August 3 piece, “LLM-driven development: the exhaustion of context switching for senior engineers, and anxiety of growth for junior engineers”. He writes that while agentic software engineering has the potential to increase the amount of software that is developed by engineering teams, it has the risk of shipping more software while understanding less about the software that is being developed. He writes that in LLM-driven development, “context transfer” becomes an expensive part of the engineering loop. The old software development rhythm of Design, Implementation, Validation, Operation (DIVO) is being compressed in implementation time in the new rhythm of design, implementation and validation in agentic software engineering.

Generative AI software development burns human context first

The old software development process had a bias towards the time it took to actually implement the software. So after the team had done a bit of design, the team spent longer writing the code for that design, then there would be code reviews to validate that the code written by the author(s) of the software was suitable for real traffic. After that the software would go live and then the team watched for operational problems, and that in turn would be used for design, implementation, validation and operation of the subsequent versions of the software. This process in software development has been referred to as DIVO, Design, Implementation, Validation, Operation.

This change in software development is important because the cheap part (implementation) and the scarce part (context) have traded places.

Implementation shrinks, but the context and validation loop gets heavier.

As we outlined before, context also becomes part of the design surface. So, within the boundaries of an agentic software engineering project, agents can only process as much context as has been previously stored in and connected between documents, tests, traces, ticket entries, comments, examples and a repository’s overall coding structure. A senior engineer later will have to actually re-construct the agent-shaped system around all those agents and contexts.

It’s an engineering culture problem with an architecture spine, therefore, the limiting resource is no longer keystrokes, but the transfer of judgment to and from the model.

The cost model finally has names

Story points have always been sort of fuzzy. Agentic software engineering makes them even fuzzier.

In the “A Cost Estimation Model for Agentic Software Engineering” paper, the ACEM authors say traditional cost models such as COCOMO II, Function Points, and Story Points assume that software cost is mainly human labor across design, coding, and testing. As software development becomes more agentic, human effort shifts toward planning, specifying, architecting, and validating agent output, while the budget absorbs LLM tokens, human-in-the-loop oversight, and orchestration infrastructure.

That framework is useful. First, it is easy to quantify the cost of token spend for an LLM. Then, the cost of the human oversight can be hand-waved away by stating that the already existing review process covers this. And last, but not least, the costs of the required infrastructure can be hidden in the agentic software engineering harness and all the additional layers such as sandboxes, eval runners, trace stores, routing layers, CI minutes, and permission layers. The improved way of speaking about the multipliers for the agentic software engineering cost estimation is by stating Revision Factors, Context Factors, and HITL Intensity Scores.

Token spend is only one layer of the agentic engineering bill.

Yes. The ACEM paper is early and has constants that are symbolic. The important part is that the authors assert that the cost estimation model presented in the paper has not yet been validated against real project data, a point worth reiterating in order to keep other people from using this work in a mistaken fashion and prematurely to estimate the costs of agentic software engineering.

Quit pretending the following costs do not exist and start measuring them inside the organization.

Also note that count of retries that change direction (as opposed to just more of the same) will be relatively high. So too will be count of review passes that find product misunderstandings (as opposed to simple syntax errors). There will be high count of time spent by engineers in explaining why a 'cleaner' abstraction actually violates an operational invariant. High count of instances where a reviewer cannot approve a change because the relevant branch does not have relevant traces, screenshots, migration plans, or rollback plans (i.e., they are not able to make sense of the change).

These are the true costs of agentic software engineering, costs that should be incorporated into the budget for agentic software engineering.

Context transfer is runtime state

Agents also use organizational state to generate code.

To my knowledge, this is the formal version of the problem description given in the paper on Context Compaction Theory. Note that in modeling tasks that are to be carried out by coding agents, model calls have bounded context windows. Therefore, agents have to compact the context states that they accumulate while working on a task. They do so by either selecting parts of the state and generating a summary or by generating a summary right away. The two games that are modeled in the above paper are a Context Selection Game and a Context Generation Game. It is then possible to relate generation to one-way communication complexity.

That sounds abstract because it is. Still useful.

Every team is already running a human version of this game. Senior engineers join conversations, ask questions like “What matters here?” and help to steer the conversation in the right direction. Reviewers decide which files to open, which logs to trust and which product edge cases smell wrong. When writing a PR description, deciding what to put in a ticket and what to delete from chat, teams are making decisions about what matters and what doesn’t.

Stale system knowledge costs. We’ve already established how documentation drift affects the use of coding agents, but the same drift to documentation affects humans in the same workflow. So an agentic coding workflow that is based on a poor representation of the current system will lead to problems, not just for the agents but for the humans too in the same workflow. The agent follows the out of date README. The reviewer has the current state of the live system in their heads. The junior engineer does a diff to find out what’s changed and learns a new rule that’s actually wrong.

This is where agentic software engineering can become uncomfortable. A team can be able to produce more code, but at the same time reduce the shared understanding of a team. The throughput graph for such a team can look great, but when the live system asks a question that no one can answer, then there is a problem.

Make context a build artifact

Package the context where the work happens.

So instead of documentation as tedious implementation detail, we turn repeated human explanations into artifacts (as mentioned above) to support the agentic engineering loop.

This is also the humane version of pairing in the LLM era. Honeycomb argues that LLM-based development moves pairing away from typing code together and toward rapid design and validation in a “rapid feedback” style, with senior engineers modeling judgment and junior engineers driving the actual implementation. That beats generating a huge number of PRs and reading comments on each of them after they’ve been put in the queue and fallen far behind. Queue-backed review comments are not mentorship just because the diff came from an agent.

A technical approach would be to code up an ‘AtumAI’ style agentic approach to generating datacenter control plane policies. Such an approach would first compile natural language goals into formal specifications of objectives, constraints, decision variables and an evaluation methodology. It would then search for solutions to the formalized problem. In evaluations of such approaches, it has been found that the policies generated by agents for control of datacenter control planes outperform the same policies that have been hand-engineered by experts in terms of various metrics for use in workload placement, for scaling of computer resources, and for management of power in datacenters in AtumAI’s evaluation.

This is a narrow test, but it illustrates an important point. In general, the more context an agent gets about the world it is acting in, the better it will become. This is particularly the case when the prompt that is given to the agent is not as good as a specification, for example. The cleverly worded prompt is not as good as the eval. The heroic reviewer of the diff is not as good as a workflow that records why a change is safe.

When a pattern repeats, promote the behavior into deterministic execution. I have previously noted that exploration should turn into straightforward coding work when the shape is known. The same thing applies to engineering culture. If a senior engineer explains the same operational invariant for the third time, stop burning senior attention on rediscovery. Put that invariant in code, tests, agent instructions, and review checks where it can be found again.

Ownership survives by becoming explicit

Agentic coding means branch sprawl and weak ownership.

The better bargain is boring. Fewer disconnected work streams. Stronger handoff artifacts. Live traces that feed back into the next prompt, test, and acceptance criterion. Review that asks whether the code meets product intent, whether it is safe to operate, and whether the work captured enough learning to make the next run cheaper.

Process evidence matters here. The ACWorld paper on agentic commerce found that final state alone can miss evaluated errors, while incomplete trajectories still hold useful signals. Its environment validates actions before shared state changes and records interactions for reproducible evaluation of ongoing transactions. Software teams have the same problem in a less exotic outfit. A green merge is a final state. The interesting evidence lives in the path: what the agent tried, what the reviewer rejected, which assumption changed, which trace proved the fix, which rollback path exists.

Software engineering with agents will reward those teams who treat the human context in their process as scarce runtime input. Thus, it should be spent where necessary i.e. where high quality judgment is required. All other situations where similar explanations are needed in the future can be saved by turning them into artifacts that can be used by the agents as well as by the human for verification.

So fast as to be worth only one line in a software engineering budget, implementation speed!

AI Agent Governance Gets Audited in the Trace | Focused Labs

Austin Vance — Sat, 01 Aug 2026 22:57:45 +0000

New delays in the AI Act mean teams will be lazy in exactly the wrong place.

The date moved, but the work did not. For those of us building agents in a regulated environment, the work of runtime proof has not changed. It is boring and specific: model, tool, policy, human in the loop, trace for receipt.

The latest update of the EU Commission on the AI Act Overview (July 31) sets the following dates for rules and duties for the development and deployment of AI: transparency rules for AI systems apply from August 2, 2026 onwards (Annex II, AI Omnibus, Commission’s 31 July update in the Commission's 31 July update); Annex III high-risk obligations enter into force on December 2, 2027 (Annex III high-risk AI, under the AI Act definitions); embedded high-risk product obligations enter into force on August 2, 2028.

Fine. The panic date shifted.

The engineering problem got more concrete.

The delay changes the calendar. It does not change the evidence architecture.

The deadline moved, the evidence burden stayed

A spreadsheet represents poor AI governance. It outlines all of the ‘agents’ i.e. processes that the AI is executing and attributes a policy owner to each of the ‘agents’ as well as outlines the associated risks. The board is told that there is a program of work going on in order to implement the required governance.

Just as a fire escape plan is useful as a map that accompanies the team on the run, not just as a document hung on a wall, a spreadsheet inventory of agents and their associated policies, risk, and board reporting ultimately becomes theater when there is no way to connect it to the actual run of business.

Connecting governance to execution path for AI systems (for providers and deployers of AI systems) is the whole game. By 2 August 2026 Article 50 obligations for transparency will apply to providers and deployers of AI systems. Public disclosure for chatbots, run-level reconstruction for agents.

What did the agent tell the person? What content did it generate? Which downstream system did it touch? Was the user aware that a machine was involved? Which runtime version made the decision?

Policy documents do not reconstruct a tool call.

A governance process that cannot see the execution path of a system cannot apply policy effectively. It is essentially applying policy blindfolded. This is why I keep coming back to the same governance shape: governance follows the execution path. The relevant policy layer is around identity, tools, state, approval, trace, and outcomes.

Agents turn compliance into an execution problem

Software governance used to be made up of a comfortable fiction. The approved code path was the behavior of the service. It accepted input and ran through deterministic logic. It might hit a database or two, but in the end it produced output. The audit record for all that activity would sit at the edge as deploy tickets, access reviews, log lines and incident reports.

Agents make that fiction expensive.

In complexity, an agent uses tools, sets input arguments and possibly asks for a human intervention. In the process of generating output for the user, it can also potentially disclose information. The same prompt will travel through different trajectories in complex systems, because the context, which was retrieved, changed, a model provider changed behavior or a conversation state crossed a threshold.

A recent post by Honeycomb goes into more detail about how to monitor AI systems for reliability, and clearly outlines how request-level context, model versions, retrieval results, and tool calls influence whether an AI system will work, and how failures in AI systems are to be analyzed and debugged within a single request inside the actual request.

The compliance version of this also holds. A clean monthly report does not tell the team why one instance of running an agent revealed salary history to the wrong people. A policy committee cannot figure out if the proper data minimization checks were made. A risk taxonomy does not establish if a human had the appropriate authority to interrupt a process.

The trace has to carry that work.

Governance has to be captured at the moment of action.

The trace is becoming the compliance surface

The record-keeping article of the AI Act matters here. Article 12 defines how high-risk AI systems shall technically allow automatic recording of events over the lifetime of the system. Such logs shall support risk identification, post-market monitoring and operation monitoring by means of traceability through traceability.

Article 12 reads like law but behaves like an architecture spec for live AI systems.

The live agent trace cannot be defined by prompt, completion and latency. The trace needs to follow the steps inside the AI agent's runtime: identity of the AI agent, delegated human user, tool used, resources touched, policy version, policy decision, evaluator result, human approval or rejection, output hash, model version, retrieval set and result status.

This turns the runtime state layer into a governance layer (as previously explored and mapped state layer where audit records accumulate) as audit records (including checkpoints, memory, trace IDs, policy decisions, and evaluator results) start to collect in an operational zone where said governance can either be searched evidence or become narrative archaeology.

C-Trace, a runtime compliance verification framework for AI agents, already looks at ways to transfer consent, purpose, data minimization and erasure as predicates over an execution trace for AI agent applications. The trace is generated by intercepting the tool invocations and model outputs of an AI application. Non-compliant actions are blocked. An audit log is created. Under perfect extraction, the system reports 0% attack success. Under 10% extractor noise, the system keeps attack success at or below 12%. The system keeps false positives at or below 16%. This is the result of an evaluation by the authors in the authors' evaluation.

The interesting part: the AI Act can be monitored in real time as the run happens.

Human oversight is a runtime event

Teams treat human oversight, and the related implementation into policy, as if it were only to do with the required staffing for implementing that policy. Name a human and assign a control.

Article 14 is more operational than the staffing slide. The Service Desk says oversight measures should enable assigned individuals to understand system capabilities and limitations, monitor operations, interpret outputs, override or reverse outputs, and interrupt the system by way of a stop button or similar procedure that brings the system to a safe state.

So that's an agent runtime. An approval is a state transition. An override is a recorded decision with actor, authority, input, output, and consequence. An interrupt is control flow.

This is where policy before the tool call comes in. Human oversight after the tool has already mutated a record is cleanup activity. Human oversight before risky activity is governance. The trace should clearly show this.

A lending agent can draft a credit recommendation. A human reviewer can approve, edit, or reject it. A healthcare triage agent can summarize patient history. A supervisor can stop an escalation. A support agent can prepare a deletion workflow. A privacy owner can verify scope before execution.

These decisions should also be captured as first-class events in the audit trail and not as after-the-fact comments in a case-management system.

Post-market monitoring is the part teams skip

The delayed high-risk dates will tempt executives to punt. Post-market monitoring has to become an operational habit before it becomes a compliance obligation.

Article 72 says providers of high-risk AI systems must establish and document a post-market monitoring system that actively and systematically collects, documents, and analyzes relevant performance data throughout the system lifetime to evaluate continuous compliance.

That phrase, throughout the system lifetime, carries the operational burden.

An agent changes after launch. A model changes after launch. A retrieval corpus changes after launch. A tool schema changes after launch. A permission boundary changes after launch (loosen up). A user cohort discovers a new path. The test cases never covered that new path. But one evaluator score means customer harm, the other score means no harm. The organization learns something new.

The important thing: post-market monitoring as a feedback loop. Trace samples from the field are used for evaluation. If the evaluation detects drift, then incidents are replayed and then used as test cases in release gates. The new runtime checks that are developed as part of the release gate process result in the agent getting safer and safer as more evidence is developed and flows in.

Runtime receipts are critical to financial workflows. A regulated workflow cannot be reduced to a summary of the overall performance of an agent. Rather, the review of such a workflow consists of a receipt containing a description of the action, the authority, the data, the approval, the outcome and the correction path.

What an audit packet contains

A full trace is too much for a reviewer to go through and a log line is too little detail to go by. A useful artifact for a reviewer is an audit packet cut from the trace.

For one agent action, that packet should contain:

Agent identity and runtime version.
Delegated user or service principal.
Session ID, trace ID, and parent action.
Tool, resource, and operation requested.
Purpose and data categories involved.
Policy version, policy decision, and reason code.
Evaluator scores or guardrail outcomes.
Human approval, edit, rejection, or interrupt event.
Output hash, result status, and downstream side effect.
Retention rule, tombstone status, and export location.

That sounds like work until the first concrete question arrives.

Note that the IETF Agent Audit Trail draft is an Internet-Draft and thus not a settled standard. However, for those interested in audit trail records from IETF software agents, this named the shape of JSON records containing the agent's identity, action taken, result of action, trust level, etc. that would form tamper-evident SHA-256 hash chains and optionally be ECDSA signed. JSONL, Syslog and CSV export are discussed as potential output formats for such records as a proposed audit format.

There's no need to wait for the standard to finish. The field names will change, but the architectural boundary will not. So, for example, if an agent is doing work that could affect people, money, access to information, employment, health, credit or regulated data, that work should create a record that can withstand a hostile read of that record.

The teams that start now get a boring advantage

The AI Act delay buys time for architecture, not time for governance-as-documentation theater.

The next year should be boring work. Firstly, put trace IDs on every run of the system. Secondly, start to version policy decisions. Thirdly, start to capture human interrupts as control-flow events, and then preserve evaluator scores. Finally, start to export action packets, write retention rules, and then above all else, keep raw sensitive data out of records unless a reviewer truly requires it to be in there. Rehearse the audit query before the audit exists.

This is the unglamorous side of ai agent governance. It will not demo well. Nobody claps for retention metadata.

But by then, the reason for the decision may have been forgotten. The team without runtime evidence has to interview the system after the fact.

AI Agent Testing Runs on Failure Traces | Focused Labs

Austin Vance — Mon, 27 Jul 2026 22:33:22 +0000

Agent tests should start from a failure under real traffic.

Teams want to re-run the exact failure in the trace to test the agent. A test case, not a screenshot, not a dashboard page, not a Slack thread of people agreeing to “watch it.”

That is where ai agent testing is heading. The useful unit of learning is no longer the benchmark score on Friday afternoon. It is the trace from a real run, turned into a verifier, owned by a team, and wired into the release path.

I don’t see the mysticalness fading much more from ai agent testing, but the interesting step in LangChain launching the Eval Engineering Skill is that the workflow now reads the repo, inspects the agent output, reads the traces, interviews the human who ran the task, and emits Harbor evals. The agent does not generate an opinion about quality. It generates a task the system can execute. The interesting part is that the agent is no longer only doing the work. It helps improve the engine that will run the next task.

I wrote this months ago: Traces collected from live agent behavior only become the basis for harness rewriting once an individual closes the loop from observed failure to change to the harness. This is fundamentally different from just trace storage, a pretty museum of weird agent behavior. Terrible for a test suite, nice for a touring show.

A score is a weak owner

Evaluations of agents have carried too much emotional weight. There is a simple number that a model score and a human score (a “judge” in the evaluation lingo) by definition get. Follow that up with a benchmark suite that scores by a few points here and there, and everyone pretends that that score corresponds in a simple manner to how an agent will actually behave under real traffic.

It does not.

Failures under real traffic have names: The agent picked the wrong tool. The retriever returned outdated policy text to the agent. Subagent mismeasured the error. Workflow failed to include required human in the loop. Agent attempted to write after a timeout and as a result duplicated side effect of the write.

An agent’s performance metric can improve in a benchmark setting while live traffic stays confusing. Conversely, a score does not say what broke. The trace says what is failing and why.

That distinction matters because ownership starts at the failure mode. LangChain’s IssueBench writeup, for instance, describes Engine processing batches of traces, then separating out the clean runs from the issue runs, and failing each of those into categories and attaching relevant traces to open issues or creating new issue cards for newly detected failures. In this way, the primary function of Engine’s input (traces) is turned into a usable stream of engineering work.

That is a healthier direction for agentic engineering. The artifact has to route. A hallucination, a silent tool error, a missing feature, and a bad recovery path belong to different owners. If the eval cannot preserve this, the team will be left with noise instead of work.

In Agent Failures Should Open Tickets, we wrote the ticketing version of this rule: recurring live-system failures ought to become owned work. In other words, trace + hypothesis + regression check. The new wrinkle is that the ticket should point at an executable eval opened by the failure.

The eval is a repo object

“eval” is still widely used to mean a spreadsheet (e.g. a balance sheet), a prompt (e.g. discussion prompt) a judge configuration (e.g. the evaluation configuration for a Harbor task). So all this feels pretty amorphous and soft.

A key distinction to keep in mind about Harbor artifacts is that they have boring form. The official Harbor task documentation describes a task directory with instruction.md, task.toml, a container environment, solution files, and tests. The core concepts documentation for Harbor says roughly the same thing: instruction, container environment, and test script.

An agent failure that matters should translate to something tangible: an instruction to reproduce the failure, the environment where it happened, the tools and constraints the agent had, a fixture or trace link, a verifier, and an owner who decides what change is allowed to make the test pass.

The trace matters when it becomes an executable artifact in the delivery loop.

This is where the primary ai agent workflow conversation needs to get concrete and this is the concrete part: the soft path (agent runs, agent answers, team observes dashboard in a web UI) vs. the useful path (live trace turns into failure cluster turned into an owned issue turned into a Harbor eval turn into a harness change turn into a rerun turned into a release gate).

Benchmarks should change the harness

The Deep Agents team is treating benchmarks similarly. In How We Benchmark Deep Agents, LangChain describes moving toward end-to-end Harbor evals for longer running tasks, then using those results to inform harness tweaks like removing todo-list middleware or slimming down the system prompt.

That is the right level of abstraction. The eval exists so evaluation steers the harness instead of decorating a model leaderboard.

This is where “ai agent workflow automation” gets tricky to use as a term. Automation conjures up images of more things running automatically. But the reality is to automate the feedback loop (agent fails, get trace, eval fails, fix on main (patches to prompts, tools, routing, context, middleware, model choice, etc.) and then the build proves the fix stayed fixed).

Deep Agents is a useful example because the harness is the product surface. A todo-list middleware seems benign until evals show that it distracts newer models or burns context. A longer system prompt looks safe until benchmarks show that it crowds out task instructions. A verifier that only checks the final answer looks fine until the trace shows that the agent skipped the required tool.

The reason to keep running evaluation after the release is that the data used for pre-release evaluation catches what the team had remembered to encode. And the traces from live evaluation catch what users, tools, permissions, and weird state transitions do to the plan after it has hit reality. Annoying. The job.

Graphs help when the loop has state

But for testing, graph-shaped agent systems make the story easier to follow. LangChain’s graph engineering essay describes LangGraph as a mix of deterministic paths and agentic steps, with nodes for work and edges for what happens next. The LangGraph Graph API shows conditional edges routing from graph state and Send fanning out work only known at runtime.

That matters because agent failures happen across transitions. The agent performed the first step correctly, then retried the wrong thing. It found the right document and sent it to the wrong specialist. It asked for approval while a sibling branch kept writing. It recovered from a tool error by losing the original intent.

A flat final-answer eval fails to capture that. A trace-linked task can assert the transition the agent took, and the verifier can check the produced artifact, issue category, approval state, absence of side effect, or route shape. That is closer to how agentic AI workflows break under real traffic.

An eval becomes useful when it has files, fixtures, a verifier, and an owner.

There are failures worth capturing in an eval artifact and failures that can wait. The rule of thumb is to capture failures that would cost customer trust, failures that recur from the same root cause, and failures that teams argue about after the fix. Everything else can probably stay in triage.

Approval is part of the workflow

GitHub’s new issue automation controls are a good adjacent signal to us. GitHub recently made a public preview available for agent automation in Issues. In this preview, the issue UI shows the rationale for label assignment, field assignment, issue type changes, closing, and assignment. Importantly, all of these approval panels, and associated automation, are merely workflow conveniences and do not map to any server-side security controls.

A release gate should test what the system enforces, not what the UI suggests. If an approval panel holds a label suggestion, test the suggestion path versus the agent directly applying the change. Test the permission boundary. Test the side-effect boundary when the workflow says rejection means zero side effects.

So, the release question is whether the trace, artifact, and verifier functioned properly under the same circumstances that previously failed.

This is why vibes are not an evaluation strategy. A person can like the demo. The dashboard looks great and the trend is up. The judge liked the final answer. The build still needs a particular failing case, and it must turn green when that failure mode is gone.

Own the loop

AI is simply another component in the larger engineering system. As Honeycomb’s CTO recently wrote in a letter to customers, AI amplifies the engineering system as it exists. Good practices get amplified. Bad practices get amplified. Engineering rigor matters more when AI enters delivery, not less.

A team with a healthy delivery system will turn traces into evals (because that is how the org already thinks, traces == evals). Failures create tests. Tests protect changes. Changes ship behind gates. Gates produce evidence.

A team without this muscle simply want a better dashboard. They’ll call out an interesting trace from time to time, rename a category here or there, add a sentence or two to a prompt. And of course the same failure will recur with a new name, dressed up in a new set of slides or a new Slack thread.

The engineering advantage is to treat every serious agent failure as a build opportunity. Mine the trace. Name the failure. Create the Harbor task. Assign an owner. Gate the release. Then use the release process to produce evidence that the fix held.

Then the agent workflow learns from real traffic instead of it being something it tries to survive.

AI Agents in Finance Live or Die by Runtime Receipts | Focused Labs

Austin Vance — Thu, 23 Jul 2026 22:13:45 +0000

The board wants to see the bottom line of the RFP. Show the receipt.

Financial services boards want to know what they get from the spend on AI in LangChain’s July post titled proving the ROI of agentic AI in financial services. In the case of agentic RFP responses, AML investigations, portfolio management workflows, and financial services compliance reviews, the work crosses documents, systems, approvals, policies, and people that are typically involved in such financial work.

One dashboard showing total tokens / uptime will not cut it.

Financial services already is stressful enough without questioning the value of AI. The data point: 68% of the financial-services respondents who used generative AI in a recent Snowflake survey from Snowflake were able to quantify a positive ROI from the technology, with 30% currently using agentic AI in live workflows. Moody's says Research Assistant users consume 60% more research and cut task completion time by 30%. As a result, financial services moves out of novelty category for agentic AI and goes straight to hard work of proving the value.

The hard part is proof.

Agentic AI ROI starts with work receipts

Agentic AI ROI quickly goes from clear to fuzzy as soon as teams start to measure the “wrong” unit. Spend is not value. Prompts are not work. A simple monthly usage graph proves activity. That’s to show a better board packet. (a pretty low bar).

The unit is the completed work item.

An RFP answer section is a work item. An AML alert disposition is a work item. A KYC exception package is a work item. A risk memo refresh is a work item with input data, a responsible agent, tool calls, model cost, human approvals, policy checks, writes to downstream objects, and so on, until a set of reviews culminates in an outcome for the item, the end result of the agent’s work. The end result has a set of receipts for the finance department (evals, traces, etc.). Each run of a work item through an agent’s tools generates one or more such end result receipts.

That is the receipt.

That connects to prior work on enterprise AI agents as runtime products. Finance needs the same owned boundary if it wants per-work-item proof.

LangChain’s finance piece names RFP processing and AML compliance monitoring as agentic use cases, then connects ROI to observability, governance, and economic measurement in the same stack. The operational discipline is making every meaningful run leave a receipt as it executes.

ROI proof is assembled while the agent runs, not after the fact in a spreadsheet.

The runtime is where the receipt gets written

Whatever live agents execute is already in full view of the runtime, so hiding all that inside engineering tools is the self-inflicted wound here.

An agent runtime handling real work knows the assistant, thread, run, input, tools, retries, costs, interrupts, stream events, and final state. LangSmith Deployment describes its Agent Server execution model around assistants, threads, and runs, with durable execution, real-time streaming, and horizontal scaling for agent workloads. The background-run API accepts metadata, context, webhooks, feedback keys, stream modes, and durability settings. That is enough surface area to attach business context before the work disappears into a trace search screen.

The receipt for a run should be created before the moment the run is created (i.e. before the first tool is called in that run’s execution).

Assign a task type to the run, attach the business object, cost center, policy, expected output contract, reviewer queue, and downstream system that the side effects will be written to. The runtime can then update the receipt as work is completed.

For AI agents in finance, the receipt becomes the control surface.

A receipt finance, risk, and engineering can read

A runtime receipt is boring on purpose.

That all gets recorded, and it is boring by design. There’s run ID and thread ID and task, data accessed, tool calls, model+token cost to run, latency to run, retries, approvals (and all the side-effect receipts), an evaluator score and a reviewer’s result, and business outcomes. For RFP sections we get to know if we’re accepting them with or without SME material re-write. For AML alerts they get escalated or closed or sent back for human review. For research memos they can be cited, rejected or re-written.

A receipt turns a trace into evidence finance, risk, and engineering can all read.

For Engineering it is debug evidence, for Risk it is authority and data lineage, for Finance it is cost per accepted work item, and for Product it is adoption tied to outcome. This receipt answers different questions for the teams and does not force each to make up its own “truth” for how well things are working.

Honeycomb has made similar observations in the past about Observability. As they put it in connect technical signals to business outcomes, because uptime, MTTR, and engineering productivity do not automatically yield a measure of what the business gets from the system. AI increases this already-substantial barrier, in that a healthy model, up service, etc. is not at all sufficient to guarantee that the agent using it will spend its time and money productively, or even at all.

AI agent cost is a runtime signal, and its useful question is whether the agent’s work item can be replayed, judged, explained, and tied to business results.

Aggregate dashboards are too late

They show spend by team, average latency, total runs, error rates, top agents, cost by model, and adoption trends. These are good to keep and to be proud of, but they are late by design, i.e. they cannot be used to prove the value of a single compliance-sensitive action.

Averages hide important details, especially when it comes to individual expensive runs and costly rework caused by high adoption. The expensive run might be valuable to investigate deeply. The cheap run might be garbage. The agent with high adoption might cause rework, whereas the agent with low adoption is the one doing the riskiest work for the firm.

The receipt is what lets the rollup make sense.

OpenTelemetry's graduation post is worth reading. Even though it’s talk of the future of telemetry and how to abstract it away from current vendor lock-in, it’s anchored firmly in current reality. OpenTelemetry is a graduated CNCF project and the maintainers point out that agent execution (running processes or containers) within agentic workflows (running workflows within workflow managers), and GenAI semantic conventions (as used by current AI and expected to be used in the future) are future workloads that OTel will be able to handle. Therefore, the receipt evidence that a run produced needs to be able to travel between all of the queues, databases, identity providers, SaaS APIs, and code review work-flows that the agent is executing in. The receipt cannot be locked in the private vocabulary of any single vendor.

Agent work volume turns measurement into architecture

Delegating work to agents as a regular part of one’s operating work is becoming operating volume, not a lab artifact. The recent Codex usage paper reported that active users grew more than fivefold in the first half of 2026. In addition, there are now more than 10% of users managing three or more agents in current active work at any time, and 26.6% of users using skills in their use of Codex for coding and other knowledge work.

Parallel execution breaks casual measurement.

The runtime needs to know what work item is currently active, which is pending approval, which handled customer data, which triggered follow-up jobs, which exceeded cost bounds, and which have side effects. A recent infinite-agent-loop paper found 68 confirmed failures across 47 projects. All of these failures can cause cost exhaustion, context growth, model denial of service, or repeated external side effects, and are caused by the iterations on the agent and its authority outpacing the system’s ability to observe its runtime behavior.

The Agent libOS paper gets the systems shape right: long-running agents require process identity, parent-child lineage, lifecycle state, explicit capabilities, human queues, checkpoints, events, and audit records. In financial-services language that becomes identity, authority, state, approval, evidence, and accountability.

Just as with money movement, spending authority moves into the runtime and therefore policy, approvals, receipts, and revocation need to be accounted for before the wallet signs the spending of tokens. The runtime should not simply spend tokens and hope finance reconstructs the value later.

Repeated work should become measured paths

The first version of the finance agent is an exploratory one. It will try out various tools, ask for more information here and there, fail evals (which it can recover from), request approval, and in the process learn which path is best.

Then the repeated paths should harden.

A weekly RFP process is not something that should stay in a loose agent loop forever. It’s a process of requirements extraction, approved-content retrieval, answers, citations, SME review, and packaging. And those should be a well defined process with well defined gates. (Even if there are also places for ambiguity).

That is the point behind agentic workflows getting less agentic. Repeated behavior of an agentic workflow should transform into an observable, doable, legible execution path (and remain agentic as soon as reality changes again). Likewise the ROI of a finance agent does not only grow as it becomes more agentic, it also grows as the agent’s work is first discovered, recorded, measured and then hardened.

And, side-effect layer. As we’ve written about before, side-effect receipts are critical to enabling features like retry, compensation, and ownership tracking when various tools are orchestrating work that causes side effects by running to update live systems. For a finance agent, this would include updating CRM records, requesting documents, opening up new case work, generating client-facing drafts, and opening up new compliance work, etc., all of which would need to be issued with an operation key and receive a corresponding receipt.

A trace in the workflow captures all the activity that occurred within the agent as it executed the workflow. A side-effect receipt, by contrast, captures the changes that the agent caused outside of the workflow, i.e., the actual side-effects.

Both belong in the ledger.

Own the ledger before the board asks

A distributed systems lecture is not what the board wants. Fair enough. They just want to know if their money has been converted into real work value.

To answer that question honestly requires a tremendous amount of engineering discipline to create runtime receipts that are dull enough to be understood by people in finance, risk, and engineering, and to attach all relevant metadata before work begins, to track all changes to authority throughout the process, to ensure that all side effects are recorded as first class facts, to tie the scores from evaluators and the results of human review to the work item in question, and to do rollups from the receipt as opposed to from the vibes that were brought to the work.

This also changes how one evaluates vendors for finance agent platforms. A platform without per-work receipts for finance, risk, and engineering to share is magic; a runtime with receipts is evidence, easier to defend when the model changes, when a workflow drifts, when a regulator asks for lineage, or when the CFO asks why the AI budget went up again.

Show the receipt.

Agent Protocols Fail at the Seams | Focused Labs

Austin Vance — Fri, 17 Jul 2026 21:45:42 +0000

The agent protocols are becoming integrated into normal software infrastructure, making integration easier. And this is making ownership blurrier.

These all make integration easier, so that’s great for product teams that want agents working together quickly. But then this becomes another failure surface in addition to specification. In this case, content flows quickly, and so do the different standards for delegation and state and all the rest. Meanwhile, ownership has traditionally moved more slowly through the org chart than that, so the mismatch becomes a problem for everybody.

Research is emerging around the integration of different agent protocols and the security of their composition. A 2026 paper, Formal Security Analysis of Agent Protocol Composition, investigates five agent protocols, finds 35 specification-level findings, backs them with 80 detailed implementation tests, and adds 30 failures found when the protocols were composed together as part of a larger system.

Finally, the years of pressuring AI agent vendors to connect together are starting to bear fruit. Work that can be done by isolated ‘agents’ is child’s play. The work of an enterprise is to cross boundaries in every direction. It has to traverse files, work items, schedules, databases, customer records in CRM systems, web pages viewed in browsers, software repositories, and APIs for internal services.

Clean locally. Sketchy globally.

Protocols standardize communication, not responsibility

Note to buyers: IBM’s useful definition says AI agent protocols standardize communication among AI agents and between agents and other system components; they do not standardize workflow coordination, execution, or optimization. Don’t make the stupid mistake of believing protocol support automatically includes control plane functionality.

MCP (Model Context Protocol) allows agents to discover tools and call them. A2A (Agent to Agent) allows independent agents to interact with each other using a standard interface. ACP (Agent Communication Protocol) allows agents to interact with each other using a REST interface (synchronous and asynchronous, streaming, stateful and stateless, online and offline, long running etc). It describes a thick grammar layer.

Grammar does not decide who can touch payroll.

The A2A 1.0 specification describes A2A in terms of three layers: the canonical data model, abstract operations, and bindings to specific protocols. Appendix A gives the enterprise pattern in miniature. An A2A client agent asks an A2A server agent to perform work. The server agent can then use tools and APIs supported by MCP. The architecture is useful for enterprise work and is exactly the direction people are moving AI agents.

ACP explicitly states the quiet part. The Agent Communication Protocol docs pitch interoperability between different frameworks, teams, and infrastructures. ACP exposes a REST API and is therefore opaque to the method calling the API. It supports long-running tasks, cancellation, streaming messages, and discovery. But as always: what looks valid as a method call can still be wrong when executed at runtime as part of a longer composed path of actions.

The risky boundary is where locally valid protocol calls become one composed path.

This is why MCP security starts after tool approval. Approving a server, registering its tools, and checking schemas only proves that a local component has a shape the client recognizes. Runtime security starts when the system asks a set of questions: What changed? What authority is being carried forward? What content entered the model’s context? What evidence will there be after the action is taken?

The failure moves through content, delegation, and authority

The AgentThread paper includes an example that is simple but should make security teams squint. One MCP server fetches attacker-controlled web content. Hidden instructions enter the model context. A second MCP server has file access. The agent reads local data and sends it back out through the first server. The paper describes the unsafe behavior as composition, implicit authority transfer, missing consent, and weak audit visibility across the agent runtime.

No single action needs to be out of the ordinary. Get a web page. Read a local file. Send the result of that.

The failure is the path.

The old framing of mcp security as a checklist around one server is too small. CoSAI’s Model Context Protocol Security whitepaper maps MCP to twelve threat categories and almost forty threats, including identity, access control, input and data boundaries, trust boundaries, lifecycle governance, logging, monitoring, and auditability. That taxonomy belongs in deployment design.

This is related to the prior post on the tool-call boundary. AI agent security happens at the tool call. But now that the tool call is a node in a larger composed route, the runtime must remember the content and the delegation that happened before calling that tool. A runtime that only checks the tool call is doing the bouncer’s work at one door, while there are tunnels in the building.

SDK conformance is not runtime safety

I like standards. I want fewer custom connectors. I want agents to connect to tools, agents, and servers without six months of mapping fields between corresponding data objects.

The AgentThread authors break down security failures at the protocol level to describe separate problems: unsafe behavior within a single protocol, missing SDK recommendations, missing deployment hardening, and unassigned responsibilities in cross-protocol cases. Ultimately, the authors find that agent-protocol insecurity stems from responsibility gaps across protocols, SDKs, and deployments, failures that occur when semantic content, delegated authority, and tool access cross protocol boundaries.

When a set of features required for securing agent-protocol interactions is distributed across three SDKs and documented in a wiki, no one ends up owning the seam.

Multi-agent systems already expose this kind of problem without the added complexity of security. We have written about how multi-agent systems break at the collaboration plane, at the claims, findings, handoffs, and shared-state level between agents. In the worst case, a lost finding turns into a security incident, and a lost authority boundary turns into a breach report.

Own the seam above the protocol stack

The fix starts by treating protocol composition as a runtime surface.

Standards committees can improve the grammar. Vendors can ship nicer SDKs. Security teams can review popular servers. Good. The runtime that joins MCP, A2A, ACP, AG-UI, browser tools, file tools, and internal APIs still has to own composition policy.

Standards define grammar. System ownership has to be assigned above them.

First, every step across protocol boundaries should leave an execution record: the principal, user or sponsor, original objective, delegated objective, source-content classification, requested tool or action, target resource, credential scope, policy decision, and resulting artifact. Call it a “ledger,” a “receipt chain,” or a “protocol-composition record.” The important thing is that the record exists for the entire composition path and is preserved for the duration of the execution.

As cross-protocol actions are executed, the authority granted to the previous step in the protocol should be restricted, such that subsequent steps in the protocol require additional justification or authorization, in line with the authority narrowdowns identified above. In particular, A2A delegation should not transfer the ambient authority of the calling principal to subsequent agents, MCP tool calls should be executed with credentials scoped to the task at hand as opposed to the human’s session, and browser-side agents should not silently inherit backend privileges as a result of a convenient protocol bridge.

Third, untrusted content needs to be labelled by the runtime as to its origin. This will allow policy to determine if later content can be used to justify actions that involve model processing of that untrusted content. A simple prompt-injection defense that only scans new text as it is injected into a model will not be effective if the injected text consists of, for example, web pages, ticket comments, email bodies, etc. Simply labelling such content as originating from the “user” will be insufficient to control its usage, since it would be allowed to be used to justify virtually any subsequent action.

Fourth, we need to carry trace context with us across protocol boundaries. As we said before, agent traces need to cross the MCP boundary because otherwise we would be left with planner traces with no tool/service spans, a useful story with a missing crime scene. But when protocol composition is involved, the downstream calls made by A2A or ACP can also cross MCP boundaries, and thus the traces they produce need to cross boundaries as well. If instead each of the different protocols produced a separate trace with no correlation between them, then any post-incident review would be pure archaeology.

Fifth, tracking of what to undo in case of failure (rollback) has to be attached to the composed path as well. The number of steps to undo can easily exceed the number of steps to complete a task already. If a tool creates a ticket, updates a file, sends out an email to customers, updates a CRM like Salesforce, and stores something in memory, then all of these steps have to be undoable as a whole. Otherwise the only thing to do in case of failure is to manually clean up the mess, which is what happens in enterprises already (classic).

The runtime is the security boundary buyers actually get

The protocol ecosystem is moving in the right direction. MCP lowered tool integration friction. A2A gives independent agents a shared way to delegate and report task state. ACP gives teams a simple HTTP path for agent interoperability. These are good tools for building useful workflows.

They are also incomplete by design.

A protocol specification describes messages, the state through which an agent goes, how an agent is authenticated, and the semantics of the operations that the agent can perform. But, the specific enterprise data structures, human rules for approval, scope of particular credentials, rules for handling an incident, and rules for rolling back actions that have been performed, all of these are outside of the scope of a protocol specification. They live in the runtime and application architecture around it.

So the procurement question should change.

Don’t just ask if the vendor supports MCP or A2A, because everybody supports the hot acronym eventually. Ask what happens when an agent delegates across A2A, calls an MCP tool, consumes untrusted content, accesses internal data, and returns an artifact to a user. Ask where the runtime’s ledger is. Ask how A2A-delegated credentials are minted for the task at hand. Ask how trace context is propagated throughout the composed protocol chain. Ask how policy sees the origin of new content introduced into a model. Ask how a system would roll back the damage done by a four-step process where each step seemed rational inside the bounds of the current protocol.

That is the AI agent security boundary now.

A protocol makes the path easier to build. The team running the system still owns what happens when authority crosses the seam.

AI Agent Infrastructure Is Splitting at the State Layer | Focused Labs

Austin Vance — Thu, 16 Jul 2026 21:40:53 +0000

The agent stack is so boring where buyers keep looking: the loop.

Loop in sight, the rest of the system is elsewhere in the loop (the agent loop did not disappear). Even more, a layer has been created to manage durable records of work done by the agent: state (a durable record of the work done by the AI agent to service customer requests). This is the crux of enterprise work. The enterprise will configure a system (AI or not) to create and maintain such records and also decide who can read such records, how long to keep them, what they are allowed to do with such records, etc. With the introduction of the state layer, AI agent infrastructure now splits at the state layer. Model + a prompt-centered agent no longer represents the infrastructure argument for AI agents.

The state layer of an AI agent is now the dividing line between the agent being a product and the agent being a parlor trick that happens to access Salesforce etc.

The loop is the wrong buying surface

The harder questions live one layer down.

Where does the state live after the tab has been closed? What writes the checkpoint? What happens when a tool call is successfully completed but the model fails before a summary can be generated? Where do traces, policy decisions and permission grants enter the run? What can an enterprise export when the vendor contract expires? Who can delete memory? Who can subpoena memory? Who is responsible for the record when two agents are working with the same customer and writing to the same file?

The durable boundary is where the agent becomes operating infrastructure.

State turns agent behavior into infrastructure.

The database is becoming the backend

On Thursday, LangChain and MongoDB announced a partnership that frames agent systems as requiring retrieval, persistent memory, operational data access, observability, and reliable deployment. The real launch is a set of tools, and the proof that the partnership matters sits in the “more about” line: "Agents need more than a model and a prompt". A tool-centered agent (or prompt-centered builder) is not a product.

That sentence is doing more work than the launch language around it.

The official LangGraph documentation notes that durable memory should use a database-backed checkpointer, with examples for Postgres, MongoDB, Redis, and Oracle. The LangSmith deployment documentation exposes the state-layer switch directly: set LS_DEFAULT_CHECKPOINTER_BACKEND to mongo and provide LS_MONGODB_URI. That line belongs in the platform review, right next to storage, deployment, and data residency.

Boring. Also the point.

State is where lock-in turns real

O'Reilly's 2026 AI agent architecture diagram defines six layers between the LLM and an AI agent running real workloads. Memory / persistent state is a first level primitive in the architecture above the vector database. This architecture matches the actual experience of the market for the framework layer: easy to switch out when the application is well bounded. State is much harder to switch out.

TGVP outlined a similar argument for how stateful services create moats because memory graphs, identity stores, policy logic, and audit history compound over time. While I dislike moat arguments for infrastructure (such as how custom UIs in enterprise software create a barrier to entry that replaces simply offering a “connector catalog” which can then be replaced by a single adapter), there is merit in the increased memory graphs, identity stores, policy logic, and audit histories that compound over time.

I wrote this out originally as part of my vendor evaluation criteria. The first part was model neutrality. This seems to be something people want to hear. Durable memory should use a database-backed checkpointer, and implementation details for that checkpointer should be exposed to review (storage, deployment, data residency, etc) long before the memory format, the trace (or log) schema, the policy and decision-making portions of the state, and the audit export. Open model routing on top of closed state is a nicer cage with better lighting.

Governance follows the state

The state layer also pulls governance out of slideware.

Agent governance can sound abstract and be confined to slideware when it is stored in policy documents. But as soon as the platform is viewed as a series of layers, the team starts to ask concrete questions about a given run of an agent. For example: who was the principal that started the run off? What was the task that narrowed down the set of permissions that were granted to that principal? What was the tool call that was made that actually crossed the boundary of what that principal was allowed to do with that set of permissions, given by that task? What was the policy that actually allowed that tool call to be made by that principal, given by that task? What was the checkpoint that actually recorded that run of the agent? And what was the trace that actually proves that that run of the agent happened in that order.

Research into the security of Agent-based systems is only just beginning to explore the boundary between what the agent is granted permission to do and what the MCP server will actually allow the agent to do. AgentBound looked at 296 popular MCP servers and found that automatically generated permission manifests worked without modification 80.9% of the time, with 0.6 ms average enforcement overhead. It is clear that MCP tool access is heading in the direction of explicit manifests and runtime enforcement of access rights rather than the current model of trust-by-default and running of host processes.

Identity work also dovetails nicely with this analysis. A 2026 AI identity report defines AI identity as a continuous relationship between what an agent declares and what it is observed to do. This relationship is established through the declaration, the observation, and the confidence in that observation. Thus the same architecture that tracks the state of a productive AI also establishes and tracks the identity of that AI. Declaration without observation is a credential. Observation without a durable store of state is a log. Confidence in the system’s understanding of the relationship between an agent’s declared identity and its observed behavior requires that the system have a durable store of state and be able to compare what has happened to what the system expected to happen.

Task-scoped access control, a topic we’ve covered in the past, also falls under this topic. A writeup for TrueFoundry’s TBAC (Task-Based Access Control) function explained how current identity-centric access models can strain under the weight of agents, because each task may need to call for a different permission set. That can happen quickly, and is particularly exposed to prompt-injection attacks. In short, TrueFoundry argues that task-based controls bundle minimal permissions for the duration of work. And that’s a great architecture, even if the vendor packaging leaves a bit to be desired.

We have looked at the authorization side before through tool-call security and agent principals (e.g. running an agent as a user). From a state-layer perspective, these all tie together: authorization, identity, memory, checkpoints, traces and audit records. The record of an agent doing work.

The stateful loop is the product

A live agent run has a rhythm.

Load memory. Call tool(s) and enforce policy for each. Write trace for each tool call. Record audit event. Update run state. Later, resume run from evidence collected during run. As mentioned previously, each live agent run will have a history which can be inspected by the platform. Such a history provides much greater value than having the agent’s runs be driven by prompts to which the model may or may not answer correctly.

State compounds.

The work does not have to be “unlearned” when an agent run is abandoned for whatever reason. A new run of work can start from the exact point at which the previous run left off.

However, all of this work would be for naught if the evidence from past runs of an agent were not properly monitored. This is why agent monitoring is infrastructure. The stateful agent loop that we discussed earlier includes a section for “write trace and audit records” for a reason. This is the means by which all of the side effects of a run of an agent are recorded, so that they can be inspected by a human (or automated system) later.

It is therefore no surprise that agent platforms that can handle real workloads are focused on platform work (as illustrated before).

The model may still be the expensive line item. The state layer is where operational trust accumulates.

Platform evaluation moves to the state layer

The buying checklist should change.

Where does it store the checkpoints for previous runs? What is the format of the platform’s memory? How does it join together the information from thread ID, run ID, trace ID, and audit ID? Does it record permission decisions made by a tool call with the record of that tool call, or does it have to rebuild that information from elsewhere? Can it export the state information on the platform surfaces, or is that something that one has to ask support to do for them? (Note that just because something is exported as a state, it doesn’t mean that the state information exported is the same as the information that was stored as state information on the platform surfaces). Can the system delete the memory of one user without affecting the memory of other users that are stored in the same index? How does the system partition out the information that it stores for state information by tenants, by environment, by region, by retention rules, etc.

Then ask the uncomfortable version: what gets lost during migration?

Prompt history and agent builder configuration are one thing. Checkpoints, memory, traces, approvals, audit records, and policy history are another. If the vendor can’t migrate all of that to the customer's new backend then the vendor is in control of the live behavior of the agent. Yes, the agent runs on the customer's cloud. Yes, the customer pays for the model and has a model key that the customer uses to sign into the agent. But the memory, the record of the agent’s governance, the record of the agent’s work, that lives somewhere else.

Own that layer or the agent's memory, governance and operating history belong to someone else's backend.

Agentic Workflows Should Get Less Agentic | Focused Labs

Austin Vance — Wed, 15 Jul 2026 03:32:07 +0000

Agentic workflows are supposed to get boring.

The first pass can be exploratory and pretty expensive. The tenth pass should be boring and not have to rediscover the solution through a completely new set of tools. That workflow has earned a promotion.

That is the useful part of the new Progressive Crystallization paper. It names a lifecycle agent teams are going to rediscover the painful way: agents explore, traces prove repeated behavior, tests turn that behavior into workflow code, and telemetry from those runs decides when the workflow gets demoted back to the agent layer.

The paper presents data from a cloud-network operations system handling real incidents in which deterministic executions went from 0% to 45% over eight months. In the meantime, the per-incident agent cost went down by more than 70% and the number of incidents doubled. The model did not simply get cheaper. Solved work stopped being agent work.

Exploration belongs upstream

We continue to see agentic workflows get treated as a fixed architecture choice: build an agent, give it tools, add approvals, add observability, and hope the loop gets better.

The better shape is a lifecycle.

The Progressive Crystallization taxonomy organizes the various possible forms execution can take in a lifecycle agent. The Type 3 form of execution is 'agent-orchestrated'. That is to say that an agent investigates a problem with bounded autonomy, checkpoints reads and routes writes through human approval. Type 2 forms of execution are 'hybrid'. In a hybrid execution a workflow of steps is owned by a workflow, but the model is used to interpret or classify information at scoped points in the workflow. Type 1 forms of execution are 'deterministic'. They consist of typed API calls, conditionals, and so on, that together form a rigid and non-negotiable execution path. There is no place in a Type 1 workflow for model invocation at runtime.

The maturity curve moves agency out of repeated execution and leaves it for novelty.

This is how enterprise teams build out agentic workflows. First, a novel incident gets explored with tools and state. Then, as that incident repeats, the team wants a workflow that implements that path of work. Then, as that workflow starts to get weird on the edge cases, the system needs a path back to exploration. And that's what we've been saying all along about LangGraph as the foundation for enterprise agents.

The compile step is the economic boundary

First we "reason" once through a problem. Then we execute that same problem solution over and over again as repeated incidents of the same thing. The Progressive Crystallization paper documents a real-world crystallization workflow, which turns out to be a straightforward incarnation of this fundamental software pattern.

The Compiled AI paper describes a class of live AI systems in which an LLM is used once to generate an artifact, that artifact is then validated, and then a large amount of code is run with zero execution tokens. The authors evaluate function-calling, reporting 96% task completion with zero execution tokens, and a break-even point of 17 transactions to compile the function. They report 57x reduction in tokens at 1,000 transactions.

If a workflow gets run repeatedly (i.e., it is a repeatable process), the fact that it is an agentic workflow gets to be a tax on the number of times that workflow gets run. That tax is enforced by runtime inference and, thus, is best countered by reducing the number of runtime inference steps, in particular the number of execution tokens required to run each step of the workflow. There are different approaches to do this (for example, model routing, prompt caching, reducing the size of the LLM used to implement each step of the workflow), all pointing at the same idea: a process repeated enough times becomes worth compiling into a function that can be called directly. We previously laid out the cost argument in AI Agent Cost Is a Runtime Signal, but that was at the level of an entire agent, whereas this tax can be incurred by individual workflows within an agent. That is the basic move behind Compiled AI.

Note, a workflow that is promoted will have a slightly different contract on the trace. The trace will serve as a candidate specification, which in the end would have to pass acceptance tests. The steps in the promoted workflow would have to have side effects that are idempotent (i.e. with keys, upserts, read-before-writes). The human approvals in the trace would have to be encoded in the workflow as explicit states instead of as mere chat turns. The failure modes of the steps in a promoted workflow would have to be of a particular type (i.e. typed exits).

LangGraph keeps the seam visible

So where does the seam go between exploration and execution?

LangGraph is especially useful in this situation because it does not force a team to decide between using a free-form agent loop, and building a full-fledged workflow engine. Instead, it lets a team use existing Python control flow and wrap API calls and other execution logic with persistence primitives, such as memory and checkpointing. Two primitives in the Functional API, @entrypoint and @task, are plain, and they end up powerful. An entrypoint defines a workflow, while a task defines work within a given workflow.

This plainness matters for two reasons. The first is that repeated execution of a particular path through the agent is just repeated execution of a function. Therefore that function can be written as Python code, complete with a model call in the middle if that still makes sense for the application. Second, the task API lets a team write one task for database records and another task for approval. In the first task, the write can carry an idempotency key, and the approval can carry its own. This way, the workflow can still do approval steps, and the workflow still gets the benefit of persisted runtime state for the agent, without doing that in a single, tricky prompt.

LangGraph also tracks changes to past workflow runs, and implements important implications for a reproducible workflow API. Saved workflow checkpoints are loaded into a resumed workflow from the beginning of the entrypoint, re-running @entrypoint calls and loading saved results from @task calls completed before the last checkpoint. The backward compatibility docs make the implication blunt: adding, removing, or reordering task calls before the last resume point can replay incorrect cached values. Nondeterministic code outside a task can also change behavior during replay.

Just like compiled code, a crystallized workflow is something that can be run again and again. It can have a version number. It can have a window of time in which to complete (a "drain window"). It can have a set of rules for migrating to a newer version in the middle of a run (a "migration plan"). And a path that has been promoted out of an agent loop is now something that behaves like API calls against a persisted record of the execution of that path. So it can be edited as a whole as a single "implementation", as a single program, rather than as a series of individual edits to individual prompts.

So the pattern here for live agentic systems, as outlined in the LangChain documentation for agent frameworks is to define a system of a combination of workflows and agents and let workflows be the simplest, cheapest, fastest and reliable means to complete a task when it doesn't require agency. Thus the mix of workflows and agents for a system will change over time.

Promotion is an evidence gate

The fastest way to ruin this workflow is to promote it based on vibes.

A model claiming a repeatable path is largely meaningless. A corpus of traces for same cases going through same path is significantly more valuable. A proposed workflow which passes tests for candidate specification is even more valuable. A promoted path of work with no human overrides, bounded by appropriate tool permissions, idempotent for all side effects, and clean rollback semantics, is starting to look like durable software.

First, approval queues belong in the runtime for agentic AI workflows. Once the path is known, agent orchestration belongs in code. Evals should sit close to where changes happen, and traces should sit close to the decision. So, the gate to "promotion" (i.e. to live usage of a particular path for a given case) is now just another release gate for a service.

The checklist of persistent state, re-executable work, idempotent writes, etc. to determine safe replay of the node is boring but there. That's what needs to be there for repeated agent paths to be 'promoted' to newly 'durable' function calls. Until then, they remain fragile in-the-loop workarounds with no persistent contract and hence no way to track, test or reproduce them for regression testing.

Observability decides demotion

Promotion gets the money conversation. Demotion keeps the system honest.

A deterministic workflow can decay. The API that a task uses can change. Firmware can be updated to add a field. A support policy can change. Workflows can continue to pass syntactic validation while slowly drifting away from the real work being done. The promoted system needs a way to notice this kind of decay without incurring the cost of the agent on every run. Traces become the control plane.

OpenTelemetry's GenAI observability post describes a single trace for an end-to-end workflow of invoking an agent, having a chat with a model, and executing a tool. Honeycomb's fast AI feedback loop writeup shows how similar telemetry can be used to drive SLOs, alerts, and dashboards, and can even be queried with Honeycomb's native query language.

This is the same reason Agent Traces Rewrite the Harness matters: traces of agentic workflow execution under real traffic matter. They become eval cases, they become harness patches, they become release gates, and they provide solid proof that fixes to solved problems actually stay fixed.

Observability turns agent traces into promotion evidence and drift into a demotion trigger.

The number of repeated incidents where the platform is still burning full agent loops, even after it has seen the pattern before, belongs in front of leadership as the waste signal.

Spend agency on novelty

The goal is not to remove agents from agentic workflows, that would be counterproductive and as silly as keeping all solved paths inside an agent loop forever.

Agents are valuable because there is always going to be uncertainty and changing reality: new requests, new incidents, misbehaving APIs, and policy exceptions between systems. The agent does all of its work at the boundary of the known process.

The operating mistake is letting that boundary stay frozen.

Workflows in live systems should decrease in agency over time. They spend fewer tokens to execute solved work. They save receipts for the execution of risky work. And they use the model for the part of the workflow where judgment still matters.

The Type 1:2:3 mix is the scoreboard. If every execution stayed Type 3 forever then the platform would be charging the business for amnesia.

Multi-Agent Systems Break at the Collaboration Plane | Focused Labs

Austin Vance — Sun, 12 Jul 2026 18:08:49 +0000

Multi-agent systems break where agents coordinate.

This investigation begins much as the last investigation began: with a run that looks workable and a seemingly adequate summary for the supervisor from three individual specialists. The trace looks busy but largely as intended, with a nice stack of model calls as specialists work individually to develop focused answers to well-specified questions from the supervisor. Then there is a human query that changes everything: who knew what when, what did they find out when, and which thing found along the way actually drove the investigation in a particular direction.

Silence.

That silence is the collaboration plane missing from the architecture.

A common architecture for multi-agent systems has a Supervisor issuing requests to Workers, each of which performs a focused task. The results are returned to the Supervisor, which synthesizes the output from all the workers into a final answer for the Human. LangChain describes this as a subagent architecture, where the main agent calls subagents as tools, keeps track of the conversation memory, and uses stateless Worker calls as a way to isolate context and run in parallel (LangChain multi-agent subagents).

Coordination is where things start producing strange results.

This operating surface, the surface of a system that is used for work or investigation, stores the information needed by a cooperative system to function: the claims made by participants, the current work of every agent, the findings so far, summaries created for other users, and event queues that store the coordination of a team of workers across turns of a conversation. Without preserving evidence of those human-visible parts of the collaboration, the system quickly devolves into vibes tracked with logs rather than cooperative problem solving.

The supervisor pattern tops out

AWS Prescriptive Guidance makes the distinction that matters. Workflow agents run through a centralized coordinator. Multi-agent collaboration uses decentralized or role-based peers that negotiate, share information, adapt, and communicate through shared memory, messaging queues, or prompt chains (AWS Prescriptive Guidance on multi-agent collaboration).

That collaboration contract looks different. Tracking a workflow means caring about the point where the coordinator delegated a task. Tracking a cooperative investigation means caring about the point where each participant made a locally reasonable move given the evidence available to that participant.

I have recently come across a clean architecture for collaboration over investigation in Honeycomb's Canvas. In their architecture each investigation is modeled as an AWS AgentCore Runtime session. Instead of each user on the investigation having an isolated chat session with a single model in that session, all the agents on the investigation can read and write to a collaboration plane, and each user has an LLM session inside that investigation session (Honeycomb's shared-canvas architecture).

The collaboration boundary belongs to the investigation, not to each user's chat.

First, this surface is straightforward to manage. Every person can follow a lead. Their agents can follow completely different leads and still work together. In the multi-user case, each user has an LLM session, and the agents for all of the users run within the collaboration plane for that investigation.

That is the boundary I care about.

The idea we have long pursued is making agents as manageable as apps, by considering them runtime products made of model calls, tooling, sandboxes, policy, traces, evals, credentials, and audit receipts (enterprise AI agents are runtime products). Multi-agent systems add another critical boundary inside that space: the shared state surface that exists across agents collaborating to complete work.

Shared memory is too blunt

A shared memory bucket hides the actual boundary.

Memory is for facts and information. Collaboration state is for work in progress. Facts can help an agent recover from confusion. Work in progress decides whether the rest of the team gets confused with it.

When an investigation is shared with other people, the collaboration plane for that investigation contains five buckets: who claimed what hypothesis, what the current actions of the agents and humans are, what has been found, a summary of what each peer found and told their human, and a set of private event queues to coordinate the next moves of the agents and humans (Honeycomb's collaboration-plane details).

The shared state is small, boring, and decisive.

Claims prevent others from duplicating work that has already begun. Current activity shows other agents currently active within the investigation and provides situational awareness, no need for each agent to re-read the entire history of conversation for a topic. Findings allow negative results to travel across an investigation. A hypothesis that has been ruled out by one agent should shrink the search space for other agents investigating similar issues. Peer summaries allow humans to coordinate without forcing each human to read the entire conversation that another human had with an agent. An event queue coordinates turns within a collaboration and prevents work from becoming polling.

Tiny data model. Large operational consequence.

The shape is also familiar. Incident teams create a shared record out of the conversations that occur during an incident. Each person claims a thread of work, pins evidence, rules things in or out, and posts a note when context changes. I wrote about the same pattern in AI incident management breaks without a shared record, because the work is shared evidence under time pressure.

Agents become manageable only when their shared working space becomes the critical boundary that gets managed.

The runtime session is an architecture choice

AWS AgentCore Runtime makes the boundary concrete. The Bedrock AgentCore Runtime docs describe a session as a dedicated microVM with isolated CPU, memory, and filesystem for that session. Runtime sessions of different users are independent. The same runtime supports MCP and A2A communication, long running workloads up to eight hours, filesystem state across stop and restart, and traces for reasoning steps, tool invocations, and model interactions (Amazon Bedrock AgentCore Runtime).

Honeycomb took that approach for Canvas and mapped investigations to runtime sessions, because investigations are the durable units of analysis. User chats are transient. Agents come and go, stall and try again, then hand back context. The runtime reflects the nature of the work.

Generic multi-agent orchestration platforms are easy to demo but difficult to deploy for real work. Counting agents is easy. The state contract is harder: what agents assert to exist, what agents can modify, what gets recorded in trace attributes, what evals check, and what humans have to follow in order to retrace the steps of the agents to a particular conclusion.

No team_of_agents.final_answer field will save that design.

Collaboration bugs do not throw exceptions

MLflow's post about multi-agent observability names three failures that become apparent when multiple agents interact in a shared environment: cascading errors, shared memory pollution, and waiting loops (MLflow on multi-agent observability).

A stale finding is a fact. A specialist can time out while the parent agent waits. A negative result may never reach another agent, so the system spends time and money re-running the same series of tool calls that the first agent already ran. A peer summary can say something misleading, like "payment errors started after the deploy," based on a finding that actually ruled out the deploy in the first place.

The trace has to expose the collaboration plane, not only the LLM calls around it.

This means that instead of simply including the text of a message in a span as the event's description, the trace needs span attributes that track coordination objects: claim.created, claim.released, finding.superseded, peer_summary.read, event_queue.delivered, hypothesis.invalidated. The attribute names are less important than the discipline. The trace can now show which agent changed the shared state of the investigation, what evidence an agent attached to a finding, who read a peer summary, what events were delivered to an event queue, and what hypothesis an agent invalidated.

Awhile back, Focused made the observability point about agentic systems that I am making today about trace semantics: agentic systems need an observable collaboration surface, not a messy tangle of disconnected logs (observability and agentic systems).

The useful version joins them.

The eval should grade the handoff

Final-answer evals are too narrow for cooperative systems. They can report that a correct answer was produced by a multi-agent system, but they will not report the bad coordination path that produced it. The same bad path will, in the end, produce a wrong answer that no one will be able to explain.

I investigated how to measure behavioral structure for cooperative work. The arXiv paper on Entropy-Based Observability for AI Agent Behavior develops trace-derived signals that give insight into agent behavior: action entropy, trajectory entropy, tool entropy, information gain, and outcome entropy. Success, reward, latency, and cost fail to capture behavior that matters. I think these metrics would be useful for getting a handle on the collaboration plane, where such behavioral structure already exists and is now observable (Entropy-Based Observability for AI Agent Behavior).

A collaboration eval should ask boring questions.

Did the agent check active claims for other agents before starting to work on a finding? Was there enough evidence added to a finding to aid agents who may later refer to that finding? Were agents treating summaries from other agents as hints or as facts that led them down a false path? Were stale findings properly superseded by new findings? Did the trace show the key event that caused the second agent to change its work plan?

Those checks belong next to the regular success metrics. They are the difference between "the system answered correctly" and "the system coordinated in a way I would trust again."

This is where conversation IDs earn their keep. A single conversation ID across agents and tools lets the trace preserve the user-facing path. Cooperative investigations also require investigation IDs, claim IDs, finding IDs, and queue-event IDs for the trace to be of any use for the work being done (AI agent observability runs on conversation IDs). The ID model must mirror the collaboration model. Correlating calls is not enough to explain the work done by the system.

Own the collaboration plane

Multi-agent architecture should start with the shared record.

First, determine the durable unit: a case, incident, investigation, branch, or task. Then decide which collaboration objects reside within that record: claims, findings, active work, peer summaries, evidence, event queues, approvals, rejected hypotheses, and rollback notes. Agent handoffs, for example, can be implemented as routing, but that would miss the point of runtime state. Agent handoffs turn routing into runtime state, and LangChain handoffs persist state across turns by updating a state variable and changing behavior based on that state (LangChain multi-agent handoffs).

What is the shared surface of truth while agents work together?

If the durable unit of work is the transcript, then the system is simply a chat app with extra workers. If the durable unit of work is a vector store, then the system can store facts about work, but it cannot do the work. If the durable unit of work is a traceable collaboration plane with explicit objects, owners, mutations, and eval checks, then the system has a shot at real work.

Buyers will specify multi agent AI, multi agent architecture, and multi-agent collaboration because those phrases sound like capability requirements. I would interpret the ask as a collaboration-plane contract which the AI agents execute.

Show the objects. Show the trace. Show the eval. Show how two agents chase a single hypothesis for a while. Show how one agent can go down a wrong path and write a finding that is bad for subsequent work. Show how a human can find that finding and correct it. Show how another agent can then change direction as a result of the new finding.

That is the system.

The agent count is trivia.

Enterprise AI Agents Are Runtime Products | Focused Labs

Austin Vance — Fri, 10 Jul 2026 17:53:03 +0000

Enterprise AI agents are runtime products. Teams get tricked thinking they bought a ‘clever model wrapper’ (a Deep Agents agent), but the real work goes on in code, network egress, credentials, traces, evals, deployment revisions, and audit receipts.

First, as we’ve already discussed, enterprise AI agents have a control plane now. But then there’s the real work of an agent as a product with a runtime boundary, that of a control plane running within a specific product boundary. It has an API. It has a release path. It has a rollback story. It has an owner who can explain what happened after a regulated workflow goes sideways.

LangChain and NVIDIA’s NemoClaw announcement for the Deep Agents Blueprint actually names the surface area, where Deep Agents Code, Nemotron 3 Ultra, and OpenShell combine into an agent system of models, harness, evals, and runtime work (LangChain and NVIDIA launch the NemoClaw Deep Agents Blueprint).

An agent becomes a runtime product when the loop, policy, credentials, and receipts sit inside one owned boundary.

Sensitive code makes the boundary obvious

Coding agents are useful for testing the permission boundaries of an agent. A coding agent reads files and writes files, runs shell commands, installs packages, runs tests, etc. The repo contains secrets-adjacent assumptions even when no secret string is actually stored there. As such, a coding agent can rapidly grant itself permission to do all sorts of damage.

This is why we found the NemoClaw Deep Agents Code post much more interesting than the open models announcement. Its permission set is exposed. We wrote yesterday that AI agent security happens at the tool call. Running them securely is important. The post spends real detail on the runtime. Coding agents run dcode inside of an OpenShell sandbox by default. Network egress is denied by default and can be approved on a request basis. Credentials never enter the sandbox. Each run of a Coding Agent can be snapshotted into a per-session audit log.

This gets at the operating question of the runtime-product frame: who owns the product that enforces the security boundary. When a network request is denied, the denial needs a home. When a package install is approved, the approval needs a release record. When a runtime product edits a migration script, the trace, eval, and diff need to prove that the resulting script is safe enough to merge.

A regulated team does not buy an ‘agent’. They buy a product surface on which an ‘agent’ can operate without breaching policy. Boring (which is the point).

The model wrapper is the small part

The enterprise stack is moving toward productized runtime agents because work around the model has become more important than the model itself. LangChain’s Enterprise Agentic AI Platform Built with NVIDIA combines LangGraph, Deep Agents, NVIDIA NIM, NeMo Agent Toolkit, OpenShell, LangSmith observability and NeMo Guardrails for live systems.

Here the crucial point is that an enterprise has a strong incentive to lock in the harness layer because that is where all the dependencies of the model are. Indeed, an enterprise can change the model route (i.e. switch from one LLMOps to another) as cost, latency, data boundary etc. change. But it cannot easily change the operating memory of an agent that is running live.

That is the same argument behind the agent harness becoming the lock-in layer. NemoClaw Deep Agents Code, announced by NVIDIA in partnership with LangChain, is benchmarked against the LangChain eval suite with Nemotron 3 Ultra achieving an aggregate score of 0.86 at a cost of $4.48 per 100,000 tokens to run compared to $43.48 for the next closest model in the suite of evaluations (LangChain and NVIDIA launch the NemoClaw Deep Agents Blueprint). The useful work of the agent is above the model, and the product owner can route coding, retrieval, review, and incident work through model routes based on quality and cost recorded by the harness.

Similarly, once the model determines the best course of action, the runtime determines what moves are possible (i.e., which tools to call and with what permission), what pauses are warranted, what narrower credentials are required to perform a move, and what audit receipts are left behind.

Schneider shows the operating model

Schneider Electric’s LLMOps with LangSmith case study is one of the cleaner enterprise proofs of how to treat LLMOps as more than just a launch point for open models to be used as a dashboard project. In the case study Schneider details the use of LLMOps with LangSmith to support 160,000 employees across 107 countries with 40 Billion euros in annual revenue. They have an AI Hub of 350 experts that have developed 60-plus agents that cover Energy, Assets and Developer Productivity.

Schneider Electric’s case study on LLMOps with LangSmith is interesting because observability is treated as product infrastructure. Schneider Electric uses one LangSmith workspace per AI product across development, QA, pre-live, and live environments (Schneider Electric LLMOps with LangSmith). In the live environment, all traces are stored in a workspace that can be revisited by developers for offline evaluation in their development datasets. This is what I call a correct approach to observability. A workspace per environment is what the org chart would want, but that would break the learning loop.

That same case study goes on to explain that “One Jo” supports 160,000 employees in 107 countries around the world (Schneider Electric LLMOps with LangSmith). Every conversation on that platform is traced. Live traces are reused to feed regression datasets that test new models and prompts for employee automation work.

The enterprise loop is boring on purpose: trace, annotate, test, revise, deploy, repeat.

This brings us to the deployment. The agents discussed above require streaming, long-term memory, human-in-the-loop, and background processing. So, LangSmith Deployment was the natural choice here (Schneider Electric LLMOps with LangSmith). However, here too, LangSmith takes a different approach to the typical, ‘one runtime per enterprise’ approach that AI tools follow. Instead, LangSmith Deployment allows each AI product to run on its own runtime stack, so the product owner can reason about data residency, latency, eval gates and rollback inside one product boundary.

That is where agentic AI implementation turns into change control. Even a prompt update or a new model route can touch sandbox images, a credential broker, tool grants, evaluators, and the other tools used by the product. Each such update is a product change, and the release record has to list it with the same seriousness as any service touching customer data.

Observability is part of the product, not a sidecar

Honeycomb’s work around its hosted MCP server is relevant to the second half of the runtime. Honeycomb has GAed its MCP server with support for BubbleUp, heatmaps, and histograms (Honeycomb MCP GA). The team there found CSV output, rather than JSON, provided something like 40% token savings for tabular tool output for evals, a tiny but relevant thing because runtime products make the next tool call easier to evaluate in terms of cost.

Honeycomb support is quite human also. Canvas and Honeycomb MCP running on top of Slack, Linear, code views, docs, and data context allows support to typically complete investigation and reach correct root cause before handing off to others for escalation (How Support Uses Honeycomb to Debug Honeycomb). The root cause identified by support is not buried under misinformation, detours, and misdirection. Support remains in control throughout the process, with the chain of evidence preserved.

As traces are stored as harness changes, this becomes a real problem only if the runtime can use them as eval cases, regression datasets, SME annotations, and release evidence.

Runtime governance is path governance

Path-dependent governance. One read is allowed. One analysis step is allowed. One outbound message is allowed. Put that together with customer data and the runtime has a different problem to solve.

The runtime-governance paper says that the behavior of AI agents is non-deterministic and path-dependent. In it, the execution path of an agent is defined as the central object for its governance. The paper Runtime Governance for AI Agents: Policies on Paths describes a policy function that takes into account the identity of an agent, the partial path that the agent has already traversed, the proposed next action, and the organizational state. It then returns a probability for the policy-violation of that next action. Prompt rules restrict the possible behavior of an agent. Static access control restricts the set of possible actions an agent can perform. However, in both cases, no restriction is placed on the sequence of allowed reads, analysis steps, and so on that are performed by an agent before the next action is added to the agent’s behavior.

Runtime governance cares about the path, not the morality of each isolated tool call.

This is why approval queues as runtime state matter. There is no value in approvals being made after the fact and written down in Slack as theater. A durable interrupt in the runtime can record the path, the approval reason, the approver, the edited payload, the final decision, and the audit receipts along the way.

A July 5 Internet-Draft proposes a security evaluation benchmark for AI agents. The Draft is individual and has no formal IETF status. Still, agent security evaluation is becoming its own benchmark category.

Own the product boundary

The product owner of an AI agent for an enterprise is close to the API contract owner for that product. The owner might sit in platform or product. Fine. The author of prompts is not enough to own the product.

This is why I like to speak of “runtime products”. There is something concrete to this term. It applies cleanly during architecture review and goes through the operating surfaces that already exist for normal software: ownership, release, data, policy, evidence and operations.

Using “agent” as a label for this work is too weak. Using “runtime product” is harder to duck and weave around. It frames the key architectural point: the operating surface is already familiar to the organization. Ownership, release, data, policy, evidence and operations.

This is the correct benchmark for evaluating an AI agent for use as an enterprise AI agent: Does it act?

Agent Traces Rewrite the Harness | Focused Labs

Austin Vance — Thu, 09 Jul 2026 17:48:47 +0000

The basic function of traces is to make the agent harness better. Until then, they are simply expensive evidence. Stacks of model calls and tool calls and retries, of rejected actions and user fixes, of the dollar amount spent in tokens. Someone has to look at them.

We have seen companies spend millions to store traces, name it “observability”, and then the same incident reappear just with prettier pictures (e.g. trace with annotations). The agent still loops, the underlying tool schema is still wrong, the prompt still forces a fixed-size answer that model forgot 3 steps ago, the evaluation still is focused on end answer instead of the long windy path that produced that answer.

The issue is the missing improvement loop.

LangChain put the current version of this plainly: traces are becoming the currency of long-horizon agent improvement. That should make every engineering leader a little uncomfortable. A live agent is already generating the dataset. The question is whether the organization has a way to turn that dataset into labels, eval cases, harness patches, release gates, and new deployments.

That loop is the new agent harness work.

A trace is a work order

Agent observability used to be about tracing and debugging (as we typically think of it) for model-based agents. But this is table stakes for engineering and operations in general. Observability for agents is actually about understanding how an agent works on a day-to-day basis, and how it arrives at a particular decision or answer.

That is table stakes. LangSmith's observability model splits the world into projects, traces, runs, and threads. So, for example, tracing out a failure in a project can be useful for the engineering team working on that project. To debug a long-horizon agent, however, that trace must be opened up to show the run-level activity, i.e. the individual calls that together make up the trace. And then the threads of conversation through traces that were initiated as multi-turn conversations must also be openable.

A user correction is a label. A rejected tool action is a label. A repeated request is a label. A long reasoning path leading to a trivial step is a label. A tool call that succeeds but produces an answer that is vague is a label. Live traces are an expensive piece of evidence, so traces of live agent work, traces from which we could learn to improve long-horizon agents, become an expensive dataset. A cost signal from such a dataset would be a trace in which heavy spend arrived at a place with no improvement. Such traces are best studied outside of finance meetings with coding agents, because coding-agent spend already belongs in the trace. Such study can produce tool descriptions, information about middleware, information about routing, information about evaluators, and so on.

The trace matters when it changes the next harness release.

LangChain is moving the loop into harness engineering

The latest innovations in the field focus on improving the model (because apparently that’s been neglected). In the Nemotron 3 Ultra playbook, LangChain kept the model fixed and changed only the harness: the system prompts, tool descriptions, and even the middleware around individual model calls and tool calls. Generation settings are left at vendor defaults. Yet even with a fixed model, bigger gains are to be had from changing the harness.

That is the part enterprises own.

One more important point. The model vendor can fine-tune the model to perform better on a task. And that is great for the vendor. But the live system has a local contract, with local tool semantics, and local policy boundaries. The local approval points, local data access rules, local latency and cost budgets, all of these things get encoded in the agent harness, and that is what live evidence proves about.

On the one hand, the harness playbook starts with evals, i.e. the same things one would fine-tune in order to improve a model via more prompt editing. But on the other hand, since harness changes against real traffic do not have learning signals, these too must go through the harness eval process, which are nothing but traces of agent behavior running through the agent’s live harness. Better Harness describes live traces as a high-throughput source of eval material. The only additional step is to tag, holdout, human review, and sample representative traces as with any other source of eval material.

And this is a perfectly reasonable thing to do: go through traces of Agent executions, cluster failure cases together, select a representative subset of those traces to test the improved harness functionality (i.e. to run those traces through the harness with the updated logic), holdout traces (if there are enough of them) aside for verification of lack of regression, go through all of those with human review for correctness of interpretation of failure, add tags or other identifying markers to the subset the team tests with to determine whether particular failure cases were resolved for particular changes to the harness, and then, as before, watch the new traces as they are produced by the system after the improved harness has gone live.

Trace mining produces the missing evaluation material

Opening one trace is trivial. But to get value out of traces, a team needs to mine a large number of them.

See work around fine-tuning a perceived-error classifier from Fireworks and LangChain on trace-mining for superior traces-for-learning in a Qwen fine-tuned judge, using chat-langchain and Fleet data. Their results found that, for the task of trace judging, a trace-mining based solution was 10 to 100 times cheaper at scale while matching or beating frontier models on the trace-judging task.

That product produces artifacts a reviewer can inspect:

labels that explain what went wrong
issue clusters that group repeated failure modes
representative examples for offline evals
holdout sets that protect generalization
monitors that reopen a problem when it returns to the foreground
evidence for deciding whether a failure opened by an agent belongs to the harness or to the model

The LangSmith Engine also maps traces to a structured development workflow through its product loop. The docs say it works from live traces to surface recurring issues, diagnose root cause, propose fixes, deploy evaluators, create dataset examples, and reopen issues if they resurface. All of this is also analogous to agent failures opening tickets for a human reviewer to inspect and debug. As such, trace review from such agents also surface named work with owners, with evidence, that can also be used to setup regression checks on the harness as well as the model.

Later that month, Focused argued that AI agent evaluation ends too early when it is restricted to pre-release experiments. That AI agent accuracy is an observability problem, evidence from live systems is required to prove that the system worked, follows naturally from this.

The agent harness changes after the trace

This post looks at how reviews of traces can be “focused” by having them end with a name for the update surface that will fix things. The kind of note that says “Model hallucinated” is essentially useless as it essentially says “we gave up here” and does nothing to actually help improve things.

The reason for tool misuse is that the tool contract (the documentation) and/or the schema and/or the return format of the tool are not correct. So it might be good to have a closer look at the field that is wrongly included in the schema, and why the returned blob by the tool cannot be used as is in the dashboard.

Looping points to the area of middleware. A cap. A checklist. State-aware interventions after the second search for the same thing. Do not rely on model to stop looping and then treat that as governance in the system prompt in the 7th paragraph below.

Bad reasoning budget: This is the issue of how teams route work to models. An inexpensive model can do classifier work. A more expensive model can do multi-step repair work. A model that is reading the same file over and over again deserves a harness intervention before the high bill becomes a personality trait of the model.

Perceived error (human judgment) that there is an error: These are situations where users may correct the AI agent, reject output, or even rephrase the same request over and over again until they get the desired response. In these scenarios, the traces should be used as training material for the next round of evaluation by the agent.

Domain gaps in the trace or issues not reproducible require fine-tuning the model after the problem has been understood from the traces generated before. Fine-tuning before resolving the root cause of a failure in a live system creates fossilized mistakes, pleasant to trip over as a well placed concrete boot.

The failure type decides which part of the agent system changes.

Day one can be boring. The trace can create a record that an engineer can trust. Which traces contributed? Which label fired? Which eval case was added? Which holdout protected the change? Which harness patch shipped? Which release gate passed? Which new traces prove the failure stayed fixed.

Those are receipts, not vibes.

AI agent lifecycle management starts here

Similar views on the role of live traces in AI agent lifecycle design come from Honeycomb in its write-up on AWS AgentCore. Canvas on AgentCore is a full fledged application that has session, deployment version, runtime state, online and offline evaluation, tool usage, and so on, all tracked for agent based applications. Honeycomb says to instrument the agent from day one.

But keeping them connected after release is a much harder problem, i.e. AI agent lifecycle management.

The trace-to-harness loop gives that lifecycle a spine.

Unless an organization has a real AI agent lifecycle management practice in place, all the work can just become folklore: “I remember we had to fix this bad trace once...”, “Okay, I added this one docstring...”, “Oh yeah, we replaced that model for that edge case...”. A week or two later nobody is able to say whether any of that actually fixed the particular failure being investigated, moved its expression to a slightly different case, or just masked it with a newer and cleaner-sounding incorrect answer.

The improved system creates a useful organizational structure around the agent, and therefore improvements to it, which is of greater value to the buyer than an agent that works in a workshop (as all AI do by now). This structure is a simple linear path from trace-evidence, via evals and so on through to harness-updates that in turn make their way through release-gates, until traces prove that changes worked or not. Conversation IDs make the trace coherent across the path, but coherence is only the start. The more valuable structure is one that shortens that path.

The winning team will have the shortest path from trace evidence to a harness release that safely changes what the agent does next time.

AI Agent Security Happens at the Tool Call | Focused Labs

Austin Vance — Wed, 08 Jul 2026 23:45:08 +0000

At what point does a team treating AI security as plumbing, meaning nobody wants to investigate, debug, or test it because that would slow the work down, fail? At the tool call.

A dangerous question that must be answered at every tool call: who is acting, which resource is canonical, what grant applies, which capability is being invoked, where the output can flow next, and what receipt exists if the runtime says no.

Connectivity to such resources is becoming easier to set up. The recent MCP SDK betas for the 2026-07-28 spec release candidate make that point plainly. So security needs to move from the setup of connection to a resource to control of execution within that resource.

The HCP paper, From Tool Connection to Execution Control, first defines eight runtime invariants that MCP-style systems need to satisfy: metadata non-authority, grant-backed approval, canonical resources, principal binding, scoped capability invocation, source-and-target data-flow authorization, deny-path audit, and explicit protocol state. For a long time these read like abstract requirements. In reality each of these is the minimum viable security invariant for an action that an agent is performing by updating CRM records, creating a customer, creating a sales process, or reporting on data in CRM. The runtime has to resolve the canonical resource, bind the principal to a grant, invoke only the scoped capability, authorize data flow, and make every denial auditable.

The approval prompt fires too early

A consent prompt tells one thing: did a human approve of this action or not. But approval of a proposed action, meaning a model asking to do something with the agent, and approval of an action after resources, policy, and tool state have resolved are two different things.

In reality, tools are not isolated containers to which an agent may make a single call. Rather, agents discover tools, read the tools’ metadata, copy data from one tool’s output to another’s input, run through loops of previously learned actions, re-run previously successful code with slightly modified parameters, and pipe the output of one tool to the input of another. As such, the place to put governance on an agent is before it makes a tool call, which is where we wrote about governance before the tool call, and after the tool call has been approved, which is where MCP security starts after tool approval. And that is narrower than it sounds, because after approval has been given to a call, the runtime is now responsible for governing the execution of that call, which is to bind the call to the particular principal, grant, resources, and capabilities that had been approved in the dialog that prompted the approval in the first place.

Now on to the HCP benchmark. The naive baseline for an MCP-like architecture allows all ten attacks to be modeled, the practice-informed mitigation strategy allows six, and the HCP benchmark blocks all ten attacks while providing complete audit evidence for the controlled benchmark in the paper. The paper itself is not something to run as a product tomorrow, but it gives a practical perspective on what prompts, metadata warnings, and approval dialogs are missing when the runtime does not own the execution objects for the calls that the agent makes.

The security decision belongs in the runtime path of the tool call.

Tool metadata cannot carry authority

Tool metadata is descriptive metadata. The model uses the tool’s metadata to choose an action. But the tool’s metadata is not authoritative for that action.

It is boring to think about tool metadata only in terms of setting up the action for the model to execute. But what if the tool’s metadata contained instructions for the model? What if a resource identifier that was provided as friendly names to the user were to map to different provider objects in the runtime? The runtime has to go ask the provider for the canonical resource for that resource identifier, and then compare that against the grants for that principal for that capability that was requested for that action. That is the set of runtime invariants for MCP-type systems as defined in the HCP paper.

Finally the MCP integration pattern transitions to the security model phase. An updateRecord CRM tool is not just another tool, even if there is already a tool with that name in the MCP workspace. It is a write tool for customer data, sales processes, reporting, and permissions. Therefore, integration of such a tool into MCP infrastructure is runtime work, which is why CRM integration becomes runtime work. Exactly the same work a user would do by pressing Save in a system of record. The only difference is that instead of a human, it is a model-guided loop, running through an integration layer.

So the simple way to look at this is that the runtime must reject the call if the following conditions are not met: the principal is missing, the grant is not found, the provider cannot canonicalize the resource, the capability is outside the scope of the grant, or the approval was for a different resolved object. Put this in a policy broker. And write down the reasons for denial.

Output handles are part of the security model

The scariest tool call is rarely the first one. It is the second one, after the agent has learned something.

When modeled data is used as input for other tools, the data can be treated as handles with owners, data classes, and checks on downstream pipes. The HCP paper investigates modes in which such data can be used, including search results, document excerpts, transcript snippets, ticket bodies, and rows in databases. Kept inside the runtime, the data can stay scoped to particular actors and capabilities instead of becoming free floating text that the agent can hand to other tools.

I’d like to provide an example of how Anthropic implemented code execution with MCP. In the example, Anthropic created a direct MCP loop between pulling a transcript from a Google Drive account and copying that transcript to update a record in a user’s Salesforce account. This would normally take 150,000 tokens, but by putting the intermediate results in the execution environment, Anthropic was able to reduce the required tokens to 2,000 tokens. Great for them, and it highlights the added security obligations of code execution.

But, if we keep the transcript in the runtime, we can then check things like: can this principal read this? Can this capability write this to Salesforce? Can this output flow to email, Slack, or external webhooks? Should the next call in the workflow see the raw text, a filtered projection of the text, or nothing at all? These are the kinds of things that belong inside AI agent authorization.

This is also where audit stops being compliance theater. Audit records for denied pipe operations should include a receipt with source handle, target capability, principal, policy version, and reason code.

Code execution makes the runtime more important

Code execution is going to win because less context and fewer round trips are worth the workflow tradeoff. The CE-MCP paper, From Tool Orchestration to Code Execution, formalizes the jump from context-coupled execution in MCP to context-decoupled code execution in MCP. The paper also introduces sixteen different attack classes that can be launched through five different execution phases of code execution.

Similarly, LangChain defines the sandbox-as-tool pattern with the agent and its state outside of the sandbox, API keys outside of the sandbox, and the execution state of the sandbox separate from the agent state. This structure is nice as it provides a natural home for the platform to manage the credentials, the policy, the trace context, and other runtime state for the generated code, model, and harness, rather than having it all in a generated code file in a harness.

Code execution saves context, but it moves the trust boundary into the runtime.

Another AI tooling fallacy: code execution is simply another loop around the model, with potentially cheaper rounds of computation. This misses the mark fundamentally. Execution has become a privileged substrate within the AI-powered end-to-end workflow, and one that cannot be governed by simple prompt-level rules. It requires a runtime policy broker to adjudicate the required authorizations along the way.

Enterprise security teams will ask for receipts

Once agents are running across employee accounts inside a company, there is real security work in running an MCP runtime. A buyer has to confirm that the runtime has implemented the functions listed in Arcade’s guide to MCP gateways, runtimes, and registries: OAuth lifecycle management, credential storage in a vault, multi-user support with respective authentication and authorization, permission intersection, auditing, policy management, enforcement, and observability.

Zuplo’s 2026 protocol-stack post separates the MCP used for agent-to-tool communication from the A2A used for agent-to-agent communication. Such a split makes sense because while MCP allows an agent to communicate with tools, A2A enables one agent to directly communicate with another agent. Gateways for cross-agent communication will enable functions such as token binding, request validation, capability filtering, and full request logging.

From an auditable decisions standpoint, we would document each decision in a detailed record which includes: principal_id, agent_id, grant_id, resource_id, capability_id, input_handle_ids, output_handle_id, policy_version, decision, reason_code, and trace_id. All of the information from the allow path and the deny path for a particular invocation, tied back to the conversation or workflow from which the invocation was invoked. We have made the same argument from the observability side: a conversation-level audit trail gives incident response a spine instead of a pile of disconnected spans.

This is where we start to put the AI agent security practices to work. If an agent is sending emails, the receipt should name the user principal, canonical mailbox, send grant, attachment handles, and policy version. If an agent created a Salesforce opportunity, then the receipt should list the same information against the opportunity as the canonical resource in that system. The same would hold for creating a new pull request.

No magic here. Just a dull and thankless job of documenting model-driven side effects.

Own the tool call

Just as we moved from unusual tool access to the normal usage of tools via MCP, now every meaningful tool call must be treated as an event for the following classes: authorization, who is allowed to make this call; data flow, what is the input to the tool and what are the handles to its output; and audit, a record of every such call.

The first part of the test is not hard: tool calls should require different policy. A read-only weather API is different from a payment API, for example. A chart generated by a model is different from an external email. The API calls should be treated differently by the runtime, and therefore by the agent, which is programmed by a human.

The test I care about is this: can an operator explain why a particular tool call was permitted or denied by the AI agent or program without relying on the transcript of the conversation as the source of truth?

If that cannot be determined beyond the model’s transcript, then the agent security program is based on manners, and it matters not whether the system uses MCP, OAuth, a nice organized catalog of approved tools, and a tidy approval prompt dialog. The real boundary is the tool call itself, and either the runtime owns that boundary and all the vagaries of the model, tool metadata, the sandbox in which the generated code runs, and the APIs that the runtime and the model and the generated code in turn call, or the runtime inherits all that.

Own the tool call. Write the receipt. Make denial a first-class path.

Coding Agent Spend Belongs in the Trace | Focused Labs

Austin Vance — Sun, 05 Jul 2026 23:26:05 +0000

Coding-agent spend gets weird the second it leaves one developer's laptop.

Here is a specific example where a single feature goes through Claude Code, Codex, Cursor, Copilot, OpenCode and a Deep Agents custom harness before finally being merged in a pull request. Each of these, as standalone tools, can be “perfectly reasonable to buy” with “local usage screen” in front of the team. But then the invoice arrives and the work actually costs something completely different, a question that the engineering lead has to answer.

That question cannot be answered from a vendor bill. It has to be answered from the run.

The problem with “Coding-Agent Bills: Feature Cost in the Workflow of Multiple Coding-Agents” was first discussed on LangChain’s blog in a July post outlines the shape of the problem. As previously mentioned, the feature can interact with several coding agents like Claude Code, Cursor or even Copilot. Each of the mentioned coding agents keep logs of their activities in various formats. As a consequence, it is very difficult to tell the cost of a feature in a workflow. LangChain says, that first a normalized trace of all activities in all root sessions through all turns of interaction of the user with the coding agents in all tool calls, etc. including all the corresponding metadata has to be generated. This trace then can be filtered by session_id, thread_id, by model or provider or even by the names of individual coding tools. The hardest part of solving the problem is the generation of this trace.

The unit is the trace.

The invoice arrives after the damage

Finance sees an invoice after it has gone through the run process. Engineering sees a loop that is still costing money.

The waste that coding-agents produce is behavioral waste. That is to say, it is the same patterns of suboptimal behavior that other agents produce. Time and again an agent will keep retrying the same failing test. Every time it goes to generate some text for a lint fix it will use the most expensive model available. A monolithic repository summary will be included in every interaction turn. Slow tools will be called to perform tasks only to be given very vague error messages. The same tool will then be called again and again until eventually another agent is called in to help debug the mess. Eventually a bill will be generated for all that was spent, which will tell the finance department how much was spent. The trace of all the calls, etc, generated while that bill was being incurred will reveal the precise sequence of tool calls that resulted in the bill having that amount.

Agent spend is a runtime signal. Coding agents make that signal less abstract. Spend gets tracked against a repository, branch, commit, pull request, developer, team, model, provider, tool, and session. That abstract finance problem scattered across product dashboards becomes an engineering problem once the fields coalesce into a single session trace.

Engineering problems can be fixed.

Many of the classic cloud cost playbook steps are still widely adopted by organizations building out FinOps practices, as laid out by the FinOps Foundation FinOps Foundation. Cost per token, highly volatile pricing for cloud compute and memory, particularly for GPU instances, as well as usual quotas on consumption of cloud resources, tagged resources to group costs by application or team, and real-time finance metrics that track to business outcomes. Coding-agent spend is typically treated as spend that runs in loops and programs, as opposed to being treated as spend that can be represented in spreadsheets. Thus a quota is only hit when spend hits a line, and a trace shows what led to that line in the first place (e.g. bad retry logic, too much context in function, missing cache hit, poor model routing).

Vendor dashboards answer local questions

Tool dashboards are useful until they become the only record.

Claude Code has local usage statistics at /usage, and Anthropic's cost guidance covers team spend limits, context compaction, model selection, MCP overhead reduction, hooks, skills, and subagent delegation. Fine operator surface for Claude Code. It still does not tell the team how much a PR cost after Claude Code, Codex, Copilot review context, and Cursor all touched it.

LangSmith Codex tracing feature for Codex tracing logs: agent turns, model metadata, token usage, tool calls, and subagent threads. OpenCode tracing OpenCode tracing logs: session root runs, assistant turns, nested tool calls, tool errors, time, attachments, subagent activity, token usage, and thread or session ID metadata for root and child sessions. Note that the Chat feature in Copilot exports OpenTelemetry spans where the invoke_agent, chat, and execute_tool spans contain total token usage, model data, and subagent context that gets propagated through tool execution. The plugin for the Codex tracing feature in LangSmith’s Codex LangSmith's Codex tracing plugin. Copilot Chat can also export OpenTelemetry spans Copilot Chat can export OpenTelemetry spans.

More importantly, the larger ecosystem is starting to generate trace-like evidence while the output of individual tools from session to session can be quite different from one another. Unlike a dashboard, this would be a normalized log of an entire coding session that makes full use of all of the single tools as well as all of the intermediate functionalities in between.

The trace is the cost unit because it joins behavior, identity, and spend.

LangSmith’s coding-agent metadata contract lists the shared fields by name instead of relying on observability to discover them. In addition to global identity fields like ls_agent_kind, ls_integration, ls_agent_runtime, thread_id, and ls_trace_schema_version, the contract lists run types: root, llm, tool, subagent, and interrupted. It also lists repo fields the runtime may expose: repo, branch, commit, working_directory, provider, model, tool, and subagent.

That contract is boring in exactly the right way. Boring fields make cost queryable. Boring fields let an engineering manager ask which repo burned spend last night, which agent runtime did it, which model got selected, which tool failed, and which team owns the pattern.

The same thing is true for spend. Agent observability runs on a stable conversation or session ID because that ID lets traces, tools, queues, APIs, evals, and incidents line up. Cost should line up on that same spine, not show up as line items with no causality.

Cost without behavior is accounting

A cost report that cannot point at behavior is just accounting with better charts.

One thing I would really like to see reported in a lot of detail, are all of the ways a coding-agent session could go over budget. That report could be a simple list of causes and fixes for each. The number for each cause would be the smoke for that area, and then a trace of the session in the room would show where the extra cost occurred.

LLM cost management is a behavior problem, not just a finance problem. Cut out the irrelevant stuff. Simple stuff should be done by cheaper models. Cache the constant stuff. Deciding which tool to use for a particular task. Fixing the docs that cause an agent to fail for a particular task. Trimming the subagent path that reads and re-reads same files for a particular task. That’s a finance system approval for a budget for this. Here’s a trace for that approved budget.

For clients of the agent platform, it matters to tie out spend on agents to engineering outcomes. A spend dashboard is only so useful if no one can connect the spend to outcomes such as did the agent make the team faster, did it create useful PRs, did it take review work off the team’s plate, did incidents go up or down, can we stop those runaway-until-breakfast sessions next time. Building a run record, not just a screenshot, is the only way to get there.

Gateway policy closes the loop

Visibility without a control boundary turns into monthly regret.

LangChain’s internal LLM Gateway rollout is an interesting example because the company tied spend to live control. Their writeup describes how one heavy coding-agent user could have spent thousands of dollars per week before anybody noticed. By putting coding-agent calls through Gateway, they set monthly, weekly, daily, and hourly budgets, then tied spend to traces, users, keys, agents, model calls, and failure modes. That is the correct sequence: see the run, fix the behavior, cap the boundary.

Documentation for the Gateway that this sits in front of is here: The LLM Gateway sits between agents / clients and model providers. It holds the secrets for the various providers, authenticates the caller with a LangSmith API key, evaluates spend and redaction policies for that caller, proxies the request to the upstream model provider, and traces the return back to LangSmith. The spend policies here Spend policies are organized by organization, workspace, API-key, or user, and can be limited to a monthly, weekly, daily, or hourly period. When a request is blocked (i.e. it would exceed budget), it returns a 402 with the spend-policy violation as trace metadata for the request that was issued. Gateway docs

A critical product decision here was to make policy violations trace events as opposed to just finance events that got blocked. But the block is part of the same evidence stream as the prompt, the model call, the tool results, the repo metadata, and the agent runtime.

Cost control works when the trace feeds the policy boundary.

This is where spending authority moves into the runtime. A coding agent is spending shared budget on behalf of a person, team, repository, and task. Policy has to sit near that action: approvals, budget caps, provider credentials, redaction, trace metadata, and issue creation.

Just setting a monthly limit is not enough. One also should consider hourly and daily limits for a single command run over night for example. User / API-key limits to catch misuse through integration paths that a team did not think to budget for. Workspace limits to stop experiments in live environments going out of control. And finally organization level caps to put a roof on all of this, where each of these limits describes a different way to define ownership.

The first version can be simple

The first version does not require a grand platform migration.

The trace should contain enough information to correlate the money spent with the work done by the engineers. The trace fields that need to be included in the trace in order to use them in the dashboards are the following trace fields that join spending with work done by engineers: session ID, user, team, repo, branch, commit, PR, agent runtime, integration, model, provider, tool name, token usage, cost, status and error. Subagent IDs can be added to the trace later when delegation starts to surface in traces. Also environment and service tags can be added to the trace later when coding-agent work is tied to runbooks or other work done by the infrastructure coding teams. All of the above fields should be included in the trace before they can be used in the dashboards.

Then add three questions to the weekly engineering review.

Which coding-agent sessions were expensive and useful?

Which sessions were expensive and stupid?

Which stupid session can be prevented next week?

Not glamorous but useful to know. This review will find out whether the Context Packs, repo documentation missing repo documentation etc are up to par. It will find out if the agent instructions prevent tool spam, if the model defaults are current. It will also find out whether people are using coding agents for work that should really be scripted instead. And then there are the useful expensive sessions. Deep code archaeology costs money and while a multi-hour migration agent may cost a lot of money for a multi-hour run of work, a clear outcome and a traceable run can still be worth it.

The control plane conversation continues. As coding agents become shared engineering infrastructure, registry, identity, policy, monitoring, cost, approvals, and retirement start to live together in the same operating estate we covered in Enterprise AI Agents Have a Control Plane Now.

Cost is part of the execution record

The cost of coding agents is not a side channel. It is part of the execution record for how software changed.

The uncomfortable truth here is that agent spend is someone’s engineering responsibility. Not in the performative sense of saying “use fewer tokens”, but in the normal operating sense of owning the fields, the trace shape, the gateway policy, the retry loops, and the review to confirm that very expensive work was worth it.

This is the operating model I trust: it does not shame developers for using agents, it does not imply that the cheapest model is the best. Spend should be managed the same way that good teams manage latency, errors, deploy risk, and incident noise. The signal should follow the execution path.

The invoice still matters. It just shows up too late to be the source of truth.