DEV Community: Austin Vance

Owning Your Intelligence Means Owning Your Implementation

Austin Vance — Thu, 22 Oct 2026 23:29:41 +0000

Harrison Chase's "Own Your Intelligence" argument was a recurring theme at Interrupt London last week, and for good reason. As AI systems become more deeply embedded in how businesses operate, organizations need control over the intelligence that makes those systems valuable: their context, behavior, evaluations, and ability to improve over time.

There's an important engineering question that follows: what does it take for an organization to exercise that ownership in practice?

A company can own its models, harnesses, traces, evals, and every line of code in an AI system. But maintaining and improving that system requires something beyond access to the underlying technology. It requires people inside the organization who understand how the system works, why it behaves the way it does, and how to change it safely.

The repository may belong to the company, while the expertise required to evolve the system still sits with a partner, a vendor, or a few people who have since moved on.

As a consultancy, we're part of that equation. We believe our role is to help companies build that expertise internally, so ownership extends beyond the technology to the capability to keep improving it. That's what we mean by owning your implementation.

The AI stack is an artifact of organizational intelligence

Harrison's "Own Your Intelligence" makes the case for controlling the systems that turn general-purpose models into intelligence specific to a business: how those systems behave, how their quality is measured, and how they improve. Making that ownership meaningful requires the engineering capability to put it into practice.

Every agent has models, a harness, context, memory, evals, traces, and feedback loops. People make decisions about all of them. Someone decides what belongs in the context. A failed trace becomes an eval because someone understands why it failed and which change might fix it. Someone restricts a tool or brings a human into a particular decision.

An open, portable stack gives organizations control over those artifacts. The ability to keep producing, evaluating, and changing them is what turns that control into lasting organizational capability.

Organizational intelligence is collective learning

C.K. Prahalad and Gary Hamel defined core competence as collective learning in 1990: skills and technologies coordinated inside the organization. That learning is distributed across people and gets embedded in routines, policies, and structures.

Organizational intelligence includes the written business: policies, contracts, trade secrets, architecture decisions. It also includes what people learn by doing the work. Which exceptions are harmless. Which customer signals change a decision. Which policies everyone quietly works around. Who gets pulled in on a weird case. Over time, an organization develops judgment about hard cases and about what to do when the written process is not enough.

Software has always served this. Business rules become code, approval paths become workflows, databases hold what no one person could. But traditional software is designed around human interpretation. A form gets validated, a claim gets recorded, a ticket gets routed, and then a person reads the notes, applies policy, handles the exception, and decides what happens next.

Agentic AI pushes into that judgment. An agent reads the notes, searches for policy, picks tools, asks for evidence, and decides whether a case follows the workflow or gets escalated.

For that to work, the organization's judgment has to be legible to software. What gets hard coded and what gets left to the model? Where does the agent have authority, and where does a person take over? That work turns organizational learning into executable behavior. Implementing the technology now requires the organization to implement parts of itself.

Open infrastructure and implementation capability work together

Open models, portable evals, and owned traces protect against a provider raising prices, retiring a model, or moving a feature to a more expensive tier. That flexibility matters. It gives organizations options as the technology changes. But exercising those options requires people who understand the system well enough to make changes safely.

A model change introduces a subtle regression in one customer journey. A trace reveals unofficial steps that were never in the process map. A new tool grants an agent more authority than the risk team anticipated. Can the internal team find the failure, diagnose it, and fix it?

If not, the organization still depends on whoever understands the harness, the evals, and the tradeoffs. The code is in the company's GitHub organization and the contract assigns ownership of the deliverables, but every time the system meets a new part of the business, someone else gets called.

Ownership of the artifacts is an important starting point. But when the ability to diagnose failures and improve the system remains external, the organization has yet to develop the full capability that ownership makes possible. And the distinction becomes clearer when you look at what happens inside an engineering team.

At a regulated wealth management firm with thousands of financial advisors, evals stopped a model upgrade that scored better on correctness but made worst-case response time much worse and changed which tools the agent selected. The tradeoff was visible before anything shipped.

Separately, a month of gateway traffic showed that most failures came from one cause: the model guessing repository names. One fix at the gateway removed that failure class for every team at once.

Neither of those was simply a model problem or a framework problem. They required implementation judgment: knowing what to measure, how to interpret the results, and where to intervene.

The people exercising that judgment were the firm's own product engineers. They had written their first evals paired with us. Now they add tools and run evals in CI on their own.

The firm didn't just gain a working system. Its engineers developed the ability to keep making that system better.

Implementation is now part of the core competence

A bank can buy a model. An insurer can buy an orchestration framework. A retailer can buy observability. Those are generic capabilities, and buying them makes sense.

But people inside the company need to be able to decompose a business process, separate the deterministic parts from the model-based parts, specify context, design evals, read traces, set authority boundaries, and change the process when the system reveals it was wrong.

These decisions aren't independent of the business. They require understanding how the organization actually operates, including the exceptions, tradeoffs, and judgment that rarely make it into formal documentation. And that understanding develops through implementation.

Every workflow sharpens the team for the next one. A production failure becomes a new eval. An unexpected tool call reveals a missing boundary. A recurring exception changes how the next workflow gets designed. Those lessons compound. The more capable the internal team becomes, the more effectively the organization can use the infrastructure it owns. It can adopt new models, change frameworks, improve performance, and extend agentic systems into new parts of the business without starting from scratch each time. Open infrastructure makes those choices possible. Internal implementation capability makes them practical.

This does not mean doing everything alone

Insourcing every tool and building an enormous internal AI group would be the wrong lesson. Owning a system doesn't mean never working with outside experts. It means being able to understand, change, and improve the system, with outside expertise used to accelerate the internal team rather than replace it. This is particularly important with agentic AI, where engineering practices are still evolving and production experience is unevenly distributed.

A partner can bring experience with architectures, evaluation strategies, observability, and failure modes that an internal team hasn't encountered yet. That expertise is valuable. But it becomes more valuable when the organization retains what it learns through the engagement.

Our pairing and feedback process exists so knowledge transfers while decisions get made, not in a handoff deck at the end. Our deprecation model means the partner's share of the work shrinks as the client team's ability grows. The goal is a live system the client team can explain, challenge, diagnose, and change. That's a different measure of success than delivering software that works on the day the engagement ends.

Own the ability to keep learning

As more organizational intelligence becomes embedded in AI systems, companies need to preserve their ability to understand, challenge, and improve how that intelligence is put to work.

Open models, portable infrastructure, owned evals, and accessible traces create the foundation for that control. Internal engineering capability makes it actionable.

These are the people who can swap models, change infrastructure, add or remove human review, adjust an agent's authority, or correct the judgment encoded in a system. And with every change, the organization gets better at building the next one. That's why owning your intelligence and owning your implementation belong together.

Own the models when it matters. Own the harness, context, memory, evals, and traces. Keep the stack portable. And build the capability to keep evolving it.

Focused Is Now a Claude Partner

Austin Vance — Thu, 22 Oct 2026 23:29:39 +0000

We're excited to announce that Focused is now a Claude partner, helping companies build, evaluate, and improve AI agents and applications with Anthropic's Claude models. Focused is a boutique software and AI engineering consultancy working alongside teams to solve complex business problems.

How Focused helps companies build with Claude

Our senior software engineers work directly with your team to translate a business problem into an implementation, make architectural decisions, and address the details that determine how a system performs outside a demo.

We help teams build Claude-powered agents, connect them to existing tools and data, and improve their performance in production. Throughout that work, we bring clear requirements, thoughtful integration, and evaluations that measure how well the system handles the tasks your business depends on.

Taking Claude applications into production

Every organization brings different systems, constraints, and ways of working. We build around those realities, collaborating closely with your engineers and the people who understand the business.

Reliability is part of development from the beginning. We define expected behavior, test the exceptions that matter, and instrument applications so teams can understand failures and assess changes as the system evolves.

Building with Claude, or need help with what's already running? Talk with our engineers.

Introducing the Focused Agent Academy

Austin Vance — Fri, 09 Oct 2026 16:35:46 +0000

The most rewarding role I’ve ever had

I’ve worked with a lot of great teams over my career, but one role has always stood out: teaching at Dev Bootcamp. Helping people move from nontechnical careers into software engineering is still the most rewarding work I’ve ever done.

A few weeks ago, I received a Slack message from Austin, our CEO, that started with, “Alright, I have a job proposition for you.”

He wanted to launch an Agent Academy at Focused: a program to hire strong senior software engineers and train them to become agent engineers. Given my background in education, he asked if I’d lead it.

My response: HELL YES!

Why an Agent Academy?

There are a lot of talented software engineers looking for their next role right now. But finding engineers with the depth of experience in agent systems needed to meet the standards we uphold for our clients takes time.

Many excellent engineers haven’t had the opportunity to build an agent yet. That doesn’t mean they’re starting from scratch. They already know how to design systems, write software, debug failures, and work through difficult problems. With the right training and hands-on experience, they can bring those skills to agent engineering.

So why train people instead of continuing the search? Because it’s a win for everyone.

Participants get paid to learn new skills and put them to work. We develop the engineers our clients need, instead of waiting for the market to catch up. And when we did the math, two months of training cost about the same as paying a recruiter.

Quite frankly, we’d rather invest that money in people. No offense, recruiters.

How the program works

The Focused Agent Academy is a paid, full-time, contract-to-hire program. The contract runs for three months, and participants who successfully complete the program will receive a full-time W-2 offer.

Instruction is delivered remotely, with participants expected to work from their local Focused office in Denver or Chicago two to three days a week.

My time at Dev Bootcamp shaped how we designed the program. The Academy is built around project-based learning: participants work through learning materials from our certification providers, then apply what they’ve learned by building.

Hands on the keyboard is how people learn. Writing the code, watching it fail, and figuring out why builds an understanding that copying and pasting can’t give you. That’s why the first two weeks are without coding agents. Before using them, participants need to understand the systems they’re building.

Weeks 1–6: Curriculum

Eval-driven agent development

Participants build agents in their editors, not notebooks, with eval-driven development at the center of the curriculum. They define the behavior they expect, build evaluation datasets, and establish a baseline they can test changes against.

As they work through retrieval, tool use, and multi-agent workflows, they use evals and execution traces to identify failures and decide what to change. Changing a prompt, model, or routing decision means measuring the effect across their test cases, including checking for regressions. Learning to make those calls is a core part of becoming an agent engineer at Focused.

What participants will build

Starting in week three, participants can use coding agents as part of their development workflow.

The initial curriculum covers:

Week 1: LLM fundamentals with the Claude client SDK
Week 2: LangGraph and LangChain
Week 3: Working with coding agents and building retrieval-augmented generation (RAG) agents
Week 4: Multi-agent architectures and Deep Agents
Week 5: Deployment and monitoring
Week 6: Security

Weeks 7–8: Capstone

Participants bring those skills together by building a production agent with evaluations, data flywheels, MCP connectors, tools, and more.

The capstone gives them more independent time to build, troubleshoot, and make decisions about their systems. During these weeks, they’ll also take the Claude Certified Architect and LangChain Certified Agent Engineer certification exams.

Weeks 9–12: Client work

Each participant joins a Focused team working on real client projects, putting what they’ve learned into practice alongside experienced engineers.

At Dev Bootcamp, I watched people from every background become software engineers. Now the industry is shifting again, and I can’t wait to help another group of engineers make the leap.

Is this for you?

If you’re a senior software engineer ready to build agent systems, we’d love to meet you. Whether you’re looking for a new challenge or finding your next role after a layoff, the Academy gives you paid time to learn, build, and put your experience to work.

Our first cohort starts November 2 in Denver and Chicago.

Apply now.

How to Evaluate Voice Agents with LangSmith | Focused

Austin Vance — Wed, 26 Aug 2026 23:50:24 +0000

A voice agent finishing a call doesn't mean it worked.

A scheduling agent can call the right tools in the right order and still book the wrong time, because nothing told it to confirm the caller's timezone. Another one can resolve the request and still make the caller sit through long pauses and repeated questions. Both calls failed. They failed differently.

LangChain's post on evaluating voice agents splits "did it work" into three questions, and that split is the useful part:

Execution. Did the agent follow its instructions, including the right tools, the right order, and the policies?
Outcome. Did the interaction get the caller what they called for?
Experience. Was the call smooth, responsive, and natural to sit through?

I wanted to see what that looks like in practice, so I built a small refund agent and ran evals against it. Six scripted calls, eleven evaluators. Everything below is either from their post or from what I hit building it.

Execution: did it follow instructions

Execution is the closest thing here to normal software testing. Did the agent behave the way it was built to behave.

For a voice agent that usually means tool invocation accuracy, policy compliance, whether it collected the required information before acting, and whether the answer was right given what it had. The final answer is only part of it. An agent can land on the right answer while calling tools it didn't need, skipping a confirmation, or reading data it had no business reading.

Use code where you can. If the expected behavior is explicit and machine-verifiable, don't pay an LLM to check it. Tool call order, argument values, required disclosures, status codes, excessive tool usage: all of that is a code evaluator.

My version of this is one function. The agent has to look the customer up before it files anything:


def verified_before_refund(run, example) -> dict:
    tool_calls = _outputs(run).get("tool_calls", [])
    if "create_refund_request" not in tool_calls:
        return {"score": 1, "comment": "No refund filed, check not applicable."}
    lookup_idx = tool_calls.index("get_customer_info") if "get_customer_info" in tool_calls else -1
    refund_idx = tool_calls.index("create_refund_request")
    return {"score": 1 if 0 <= lookup_idx < refund_idx else 0, "comment": f"tool_calls={tool_calls}"}

That's fast, free, and it gives the same answer every time. There's no reason to hand it to a judge.

Use an LLM judge when meaning matters. Whether the agent explained the next step, recognized an ambiguous request, or followed a policy written in prose. Keep the task narrow and the rubric specific. "Was this a good response" is not an evaluator. "Pass if the agent confirms date, time, and timezone before calling the booking tool, fail if any field is missing or the booking happens first" is.

Outcome: did it get the caller what they wanted

An agent can execute perfectly and still fail the person on the phone.

Back to the scheduling example. It collects a date and time, checks availability, books it. The workflow never mentions timezone, so the appointment lands in the wrong hour. The agent followed instructions. The instructions weren't enough.

An outcome evaluator asks whether the request was resolved, whether the task was completed rather than described, whether the fallback was right when it couldn't be completed, and whether the agent had the context and tool access it needed.

An LLM judge can read that off the conversation. When a business system knows the answer, use the business system instead.

Workflow	What to verify downstream
Scheduling	Correct person, time, location, and timezone
Customer support	Resolution status and case reopen rate
Transfers	Correct destination and successful connection
Sales or onboarding	Completed next step or meeting

The agent saying "you're all set" isn't proof. The ticket is.

Experience: was the call smooth

This is the part that transcript-only evaluation misses, and it's where most of my time went.

Don't roll latency into one number. Trace each stage: time to first audio, speech-to-text, model time to first token, tool latency, text-to-speech. Measure P50, P95, P99. The median tells you what a normal call feels like. The tail is what makes an agent feel unreliable.

One awkward pause could be transcription, the model, a slow tool, or audio generation. An aggregate number can't tell you which.

I hit this immediately. My first responsiveness number was garbage because it included the ElevenLabs call, so a slow reply and slow audio looked identical. Timing them apart took two lines:


started = time.monotonic()
result = agent.invoke({"messages": messages})
reply_secs = time.monotonic() - started      # model only

tts_started = time.monotonic()
speak(agent_reply, out_path=audio_path)
tts_secs = time.monotonic() - tts_started    # audio only

Across six calls the model averaged 3.72s per reply. TTS added another 2.7 to 4.1s on top. Those are different problems with different fixes, and one number would have hidden that.

If you're grading the voice, listen to the voice. A transcript tells you the wording was clear. It can't tell you the agent mispronounced a name, sped up, changed volume, or sounded robotic.

This is the whole reason I went down this road. The transcript seems fine when reading it, but I know there's pauses in the response.

So the audio judge gets the actual generated speech. Claude doesn't take audio input yet, so that one runs on OpenAI's gpt-audio:


response = client.chat.completions.create(
    model="gpt-audio",
    modalities=["text"],
    messages=[{"role": "user", "content": [
        {"type": "text", "text": prompt},
        {"type": "input_audio", "input_audio": {"data": audio_b64, "format": "mp3"}},
    ]}],
)

I split its output into two scores. Clarity covers pronunciation, volume consistency, and awkward pauses. Naturalness covers pacing, tone, expressiveness, robotic repetition, and whether the delivery fit the moment. Both come from one judge call per turn, cached, so the second metric is free.

Look for conversational friction. Requests for repetition, clarification loops, interruptions and overlapping speech, long silences, abnormal call duration, early termination, failure to recover after being cut off.

Context matters. A clarifying question isn't automatically friction. It's friction when the agent asks for something the caller already gave, or keeps asking the same thing.

I made this a judge that returns a count plus the names of the signals that fired, so a score can be audited:


class FrictionGrade(TypedDict):
    reasoning: Annotated[str, ..., "Which signals fired and where"]
    friction_events: Annotated[int, ..., "Total count of friction events"]
    signals: Annotated[list[str], ..., "caller_asked_agent_to_repeat, "
        "agent_asked_caller_to_repeat, repeated_clarification_loop, ..."]

Some signals in their list I couldn't score at all: interruptions, overlapping speech, and long silences.

All three need the same thing, which is both sides of the call on one clock. You need to know the caller started talking at 4.2s while the agent kept going until 4.9s, so those 700ms were overlap. Or that the agent stopped at 6.1s and the caller didn't start until 9.4s, so somebody sat through three seconds of dead air. Without both streams timestamped against a shared timeline, none of that exists as data to score.

I've got two runtimes and only one of them can produce it. The live agent runs through Pipecat on a duplex transport, mic in and speakers out, with Silero VAD deciding when a turn ends. That side knows when both parties started and stopped talking. The eval harness doesn't use it. It replays scripted caller turns through the text agent and synthesizes each reply on its own, so there's no caller audio at all, just strings I typed. No second speaker means no timeline.

So I left them unscored rather than invent a number, and the judge prompt tells it not to count them. Ask a judge to score interruptions off a transcript and it'll give you a number every time. The number is fiction.

Closing that gap means running the evals through the Pipecat pipeline instead of around it. Synthesize the caller's turns as audio, play them into the transport on a schedule with deliberate barge-ins, and read overlap and silence off the VAD events both sides already emit. Turn tracking is on already for the tracing, so the events exist. I haven't wired the eval to drive them yet.

Match the evaluator to the claim

Their post has a table for this and it's the right way to think about it:

Method	Best for
Code evaluators	Explicit, machine-verifiable behavior
LLM judges	Narrow semantic criteria with a clear rubric
Audio-aware LLM judges	Properties only audible in the recording
Business-system checks	Real-world outcomes

A transcript can't prove pronunciation was clear. A judge is wasted on something a code check settles. A call that sounds successful doesn't prove the meeting got booked.

What tripped me up

Two things I'd tell anyone starting this.

The friction judge disagrees with itself. I ran the same six scenarios twice. The scripted repeat request got caught both times, since the caller literally says "Sorry, can you say that again?" The softer calls moved around: one scenario went from 2 events to 1, another from 0 to 1. If I keep going with this I'll run each scenario a few times and look at the spread.

Report no data as no data. ElevenLabs ran out of credits partway through a run, so a batch of calls had no audio. My evaluator averaged an empty list and reported naturalness as 0, which reads as the worst voice you've ever heard. It was a billing problem. It returns None now with a comment saying why.

What it looks like in LangSmith

Every turn breaks out stt, llm, and tts as its own span with its own timing, so when a reply felt slow I could see which one was slow. The feedback scores hang off the same run, which means a regression points at a specific metric.

One caveat if you go looking right after a run: the per-run cells populate immediately but the average row lags about a minute. I spent a while convinced my evaluators hadn't attached before I checked the API and found all 66 feedback rows sitting there.

One score hides the tradeoffs

A prompt change can improve instruction adherence and lower resolution. A faster model can cut latency and get worse at recovering from interruptions. Roll it into a single quality number and you can't see any of that.

Track execution, outcome, and experience separately.

Benchmarking Gemini, Claude, and OpenAI for Structured Document Extraction

Austin Vance — Fri, 14 Aug 2026 02:51:34 +0000

A few fellow engineers and I were talking about field extraction the other day. They were both saying Gemini's been surprisingly good at pulling structured data off documents, which surprised me, since that's not really the model I'd have reached for first. So we ran an experiment using LangSmith.

I printed ten copies of a form with the usual set of fields, names, dates, IDs, a checkbox, a signature line, and asked my coworker to help me fill them out by hand. I ran OCR on them too, mostly out of curiosity. It was pretty bad, which wasn't unexpected, especially on checkboxes.

Testing it

I set up a LangGraph graph with one node that does the extraction, and passed in which provider to use as runtime context, so the eval could run the exact same graph once per model:


def _extract(state: ExtractionState, runtime: Runtime[ExtractionContext]) -> dict:
    extractor = EXTRACTORS[runtime.context.provider]
    return {"result": extractor.extract(state["image_path"])}

I hand-labeled ground truth for every form and ran it all through LangSmith so I could score each model against the exact same data.

Scoring took a bit of tuning, mostly because a plain exact string match evaluator will ding a model for writing "Sr" instead of "Sr." or "management" instead of "Management," which isn't actually wrong, it's just formatting. So everything got normalized first (lowercased, trailing punctuation stripped, dates parsed into one format, phone numbers stripped down to digits) and scored three ways: fraction of fields correct for partial credit, whether the whole record was perfect for a stricter all-or-nothing exact match, and field accuracy per dollar.

Model	Field accuracy	Exact match	Perfect records	Field accuracy / cost
Claude (opus-5)	0.934	0.364	4/11	31.8
Claude (haiku-4-5)	0.636	0.091	1/11	207.2
Gemini (3.1-pro-preview)	0.934	0.636	7/11	93.1
Gemini (flash-lite-latest)	0.884	0.455	5/11	1084.8
OpenAI (gpt-5.6-sol)	0.860	0.455	5/11	50.6
OpenAI (gpt-5.4-nano)	0.620	0.091	1/11	814.4

Claude and Gemini ended up tied on average field accuracy, but Gemini still got a lot more whole forms completely right, which is a different thing to be measuring than it sounds like at first. The other thing that jumped out was the cheap tiers. Gemini's Flash Lite scored higher on field accuracy than OpenAI's flagship did, while Claude's Haiku and OpenAI's nano both fell off hard.

Picking a metric

No single model won across the board, and that's kind of the whole point. Field accuracy, exact match, and cost efficiency measure different things, so a model that dominates one of them can easily be the weakest on another. What's nice about having an eval running is that switching which one you care about is just a matter of changing what you sort by.

Optimizing for exact match

A form that's 11 out of 12 fields right still needs someone to go fix that twelfth field, and that costs about the same as fixing a form that's only 6 out of 12 right, since either way a person has to open it back up. That's the case for caring about exact match instead of average field accuracy.

exact match and field accuracy per model with a baseline selected for exact match

Optimizing for field accuracy

Field accuracy gives partial credit for every field that's right, which matters more than exact match when a human is going to check every field anyway. Fewer wrong fields overall beats fewer broken records when the correction happens field by field instead of record by record.

exact match and field accuracy per model with a baseline selected for field accuracy

Optimizing for cost

Cost efficiency matters most at volume, since a cheaper model that's a little less accurate can still get more forms extracted correctly per dollar than a pricier one that's only a little better.

exact match, field accuracy, and accuracy per dollar with a baseline selected for accuracy per dollar

So Chris and Mike were right, though the answer splits three ways. Gemini Pro won exact match and tied Claude Opus on field accuracy, and Gemini Flash Lite won on cost. Flash Lite is the one I didn't see coming, since it beat OpenAI's flagship on field accuracy too, at a tiny fraction of the price.

Technical References:

Benchmarking LLMs for Structured Data Extraction GitHub Repo

Agentic Software Engineering Makes Human Context the Budget

Austin Vance — Tue, 04 Aug 2026 17:37:06 +0000

Agentic software engineering is moving the bill into human context.

Code is coming faster, pull requests are piling up faster and validation work lands in ever larger piles of judgment work that still need to be handled by people.

This feeling has been captured by Honeycomb’s Douglas Soo in his August 3 piece, “LLM-driven development: the exhaustion of context switching for senior engineers, and anxiety of growth for junior engineers”. He writes that while agentic software engineering has the potential to increase the amount of software that is developed by engineering teams, it has the risk of shipping more software while understanding less about the software that is being developed. He writes that in LLM-driven development, “context transfer” becomes an expensive part of the engineering loop. The old software development rhythm of Design, Implementation, Validation, Operation (DIVO) is being compressed in implementation time in the new rhythm of design, implementation and validation in agentic software engineering.

Generative AI software development burns human context first

The old software development process had a bias towards the time it took to actually implement the software. So after the team had done a bit of design, the team spent longer writing the code for that design, then there would be code reviews to validate that the code written by the author(s) of the software was suitable for real traffic. After that the software would go live and then the team watched for operational problems, and that in turn would be used for design, implementation, validation and operation of the subsequent versions of the software. This process in software development has been referred to as DIVO, Design, Implementation, Validation, Operation.

This change in software development is important because the cheap part (implementation) and the scarce part (context) have traded places.

Implementation shrinks, but the context and validation loop gets heavier.

As we outlined before, context also becomes part of the design surface. So, within the boundaries of an agentic software engineering project, agents can only process as much context as has been previously stored in and connected between documents, tests, traces, ticket entries, comments, examples and a repository’s overall coding structure. A senior engineer later will have to actually re-construct the agent-shaped system around all those agents and contexts.

It’s an engineering culture problem with an architecture spine, therefore, the limiting resource is no longer keystrokes, but the transfer of judgment to and from the model.

The cost model finally has names

Story points have always been sort of fuzzy. Agentic software engineering makes them even fuzzier.

In the “A Cost Estimation Model for Agentic Software Engineering” paper, the ACEM authors say traditional cost models such as COCOMO II, Function Points, and Story Points assume that software cost is mainly human labor across design, coding, and testing. As software development becomes more agentic, human effort shifts toward planning, specifying, architecting, and validating agent output, while the budget absorbs LLM tokens, human-in-the-loop oversight, and orchestration infrastructure.

That framework is useful. First, it is easy to quantify the cost of token spend for an LLM. Then, the cost of the human oversight can be hand-waved away by stating that the already existing review process covers this. And last, but not least, the costs of the required infrastructure can be hidden in the agentic software engineering harness and all the additional layers such as sandboxes, eval runners, trace stores, routing layers, CI minutes, and permission layers. The improved way of speaking about the multipliers for the agentic software engineering cost estimation is by stating Revision Factors, Context Factors, and HITL Intensity Scores.

Token spend is only one layer of the agentic engineering bill.

Yes. The ACEM paper is early and has constants that are symbolic. The important part is that the authors assert that the cost estimation model presented in the paper has not yet been validated against real project data, a point worth reiterating in order to keep other people from using this work in a mistaken fashion and prematurely to estimate the costs of agentic software engineering.

Quit pretending the following costs do not exist and start measuring them inside the organization.

Also note that count of retries that change direction (as opposed to just more of the same) will be relatively high. So too will be count of review passes that find product misunderstandings (as opposed to simple syntax errors). There will be high count of time spent by engineers in explaining why a 'cleaner' abstraction actually violates an operational invariant. High count of instances where a reviewer cannot approve a change because the relevant branch does not have relevant traces, screenshots, migration plans, or rollback plans (i.e., they are not able to make sense of the change).

These are the true costs of agentic software engineering, costs that should be incorporated into the budget for agentic software engineering.

Context transfer is runtime state

Agents also use organizational state to generate code.

To my knowledge, this is the formal version of the problem description given in the paper on Context Compaction Theory. Note that in modeling tasks that are to be carried out by coding agents, model calls have bounded context windows. Therefore, agents have to compact the context states that they accumulate while working on a task. They do so by either selecting parts of the state and generating a summary or by generating a summary right away. The two games that are modeled in the above paper are a Context Selection Game and a Context Generation Game. It is then possible to relate generation to one-way communication complexity.

That sounds abstract because it is. Still useful.

Every team is already running a human version of this game. Senior engineers join conversations, ask questions like “What matters here?” and help to steer the conversation in the right direction. Reviewers decide which files to open, which logs to trust and which product edge cases smell wrong. When writing a PR description, deciding what to put in a ticket and what to delete from chat, teams are making decisions about what matters and what doesn’t.

Stale system knowledge costs. We’ve already established how documentation drift affects the use of coding agents, but the same drift to documentation affects humans in the same workflow. So an agentic coding workflow that is based on a poor representation of the current system will lead to problems, not just for the agents but for the humans too in the same workflow. The agent follows the out of date README. The reviewer has the current state of the live system in their heads. The junior engineer does a diff to find out what’s changed and learns a new rule that’s actually wrong.

This is where agentic software engineering can become uncomfortable. A team can be able to produce more code, but at the same time reduce the shared understanding of a team. The throughput graph for such a team can look great, but when the live system asks a question that no one can answer, then there is a problem.

Make context a build artifact

Package the context where the work happens.

So instead of documentation as tedious implementation detail, we turn repeated human explanations into artifacts (as mentioned above) to support the agentic engineering loop.

This is also the humane version of pairing in the LLM era. Honeycomb argues that LLM-based development moves pairing away from typing code together and toward rapid design and validation in a “rapid feedback” style, with senior engineers modeling judgment and junior engineers driving the actual implementation. That beats generating a huge number of PRs and reading comments on each of them after they’ve been put in the queue and fallen far behind. Queue-backed review comments are not mentorship just because the diff came from an agent.

A technical approach would be to code up an ‘AtumAI’ style agentic approach to generating datacenter control plane policies. Such an approach would first compile natural language goals into formal specifications of objectives, constraints, decision variables and an evaluation methodology. It would then search for solutions to the formalized problem. In evaluations of such approaches, it has been found that the policies generated by agents for control of datacenter control planes outperform the same policies that have been hand-engineered by experts in terms of various metrics for use in workload placement, for scaling of computer resources, and for management of power in datacenters in AtumAI’s evaluation.

This is a narrow test, but it illustrates an important point. In general, the more context an agent gets about the world it is acting in, the better it will become. This is particularly the case when the prompt that is given to the agent is not as good as a specification, for example. The cleverly worded prompt is not as good as the eval. The heroic reviewer of the diff is not as good as a workflow that records why a change is safe.

When a pattern repeats, promote the behavior into deterministic execution. I have previously noted that exploration should turn into straightforward coding work when the shape is known. The same thing applies to engineering culture. If a senior engineer explains the same operational invariant for the third time, stop burning senior attention on rediscovery. Put that invariant in code, tests, agent instructions, and review checks where it can be found again.

Ownership survives by becoming explicit

Agentic coding means branch sprawl and weak ownership.

The better bargain is boring. Fewer disconnected work streams. Stronger handoff artifacts. Live traces that feed back into the next prompt, test, and acceptance criterion. Review that asks whether the code meets product intent, whether it is safe to operate, and whether the work captured enough learning to make the next run cheaper.

Process evidence matters here. The ACWorld paper on agentic commerce found that final state alone can miss evaluated errors, while incomplete trajectories still hold useful signals. Its environment validates actions before shared state changes and records interactions for reproducible evaluation of ongoing transactions. Software teams have the same problem in a less exotic outfit. A green merge is a final state. The interesting evidence lives in the path: what the agent tried, what the reviewer rejected, which assumption changed, which trace proved the fix, which rollback path exists.

Software engineering with agents will reward those teams who treat the human context in their process as scarce runtime input. Thus, it should be spent where necessary i.e. where high quality judgment is required. All other situations where similar explanations are needed in the future can be saved by turning them into artifacts that can be used by the agents as well as by the human for verification.

So fast as to be worth only one line in a software engineering budget, implementation speed!

AI Agent Governance Gets Audited in the Trace | Focused Labs

Austin Vance — Sat, 01 Aug 2026 22:57:45 +0000

New delays in the AI Act mean teams will be lazy in exactly the wrong place.

The date moved, but the work did not. For those of us building agents in a regulated environment, the work of runtime proof has not changed. It is boring and specific: model, tool, policy, human in the loop, trace for receipt.

The latest update of the EU Commission on the AI Act Overview (July 31) sets the following dates for rules and duties for the development and deployment of AI: transparency rules for AI systems apply from August 2, 2026 onwards (Annex II, AI Omnibus, Commission’s 31 July update in the Commission's 31 July update); Annex III high-risk obligations enter into force on December 2, 2027 (Annex III high-risk AI, under the AI Act definitions); embedded high-risk product obligations enter into force on August 2, 2028.

Fine. The panic date shifted.

The engineering problem got more concrete.

The delay changes the calendar. It does not change the evidence architecture.

The deadline moved, the evidence burden stayed

A spreadsheet represents poor AI governance. It outlines all of the ‘agents’ i.e. processes that the AI is executing and attributes a policy owner to each of the ‘agents’ as well as outlines the associated risks. The board is told that there is a program of work going on in order to implement the required governance.

Just as a fire escape plan is useful as a map that accompanies the team on the run, not just as a document hung on a wall, a spreadsheet inventory of agents and their associated policies, risk, and board reporting ultimately becomes theater when there is no way to connect it to the actual run of business.

Connecting governance to execution path for AI systems (for providers and deployers of AI systems) is the whole game. By 2 August 2026 Article 50 obligations for transparency will apply to providers and deployers of AI systems. Public disclosure for chatbots, run-level reconstruction for agents.

What did the agent tell the person? What content did it generate? Which downstream system did it touch? Was the user aware that a machine was involved? Which runtime version made the decision?

Policy documents do not reconstruct a tool call.

A governance process that cannot see the execution path of a system cannot apply policy effectively. It is essentially applying policy blindfolded. This is why I keep coming back to the same governance shape: governance follows the execution path. The relevant policy layer is around identity, tools, state, approval, trace, and outcomes.

Agents turn compliance into an execution problem

Software governance used to be made up of a comfortable fiction. The approved code path was the behavior of the service. It accepted input and ran through deterministic logic. It might hit a database or two, but in the end it produced output. The audit record for all that activity would sit at the edge as deploy tickets, access reviews, log lines and incident reports.

Agents make that fiction expensive.

In complexity, an agent uses tools, sets input arguments and possibly asks for a human intervention. In the process of generating output for the user, it can also potentially disclose information. The same prompt will travel through different trajectories in complex systems, because the context, which was retrieved, changed, a model provider changed behavior or a conversation state crossed a threshold.

A recent post by Honeycomb goes into more detail about how to monitor AI systems for reliability, and clearly outlines how request-level context, model versions, retrieval results, and tool calls influence whether an AI system will work, and how failures in AI systems are to be analyzed and debugged within a single request inside the actual request.

The compliance version of this also holds. A clean monthly report does not tell the team why one instance of running an agent revealed salary history to the wrong people. A policy committee cannot figure out if the proper data minimization checks were made. A risk taxonomy does not establish if a human had the appropriate authority to interrupt a process.

The trace has to carry that work.

Governance has to be captured at the moment of action.

The trace is becoming the compliance surface

The record-keeping article of the AI Act matters here. Article 12 defines how high-risk AI systems shall technically allow automatic recording of events over the lifetime of the system. Such logs shall support risk identification, post-market monitoring and operation monitoring by means of traceability through traceability.

Article 12 reads like law but behaves like an architecture spec for live AI systems.

The live agent trace cannot be defined by prompt, completion and latency. The trace needs to follow the steps inside the AI agent's runtime: identity of the AI agent, delegated human user, tool used, resources touched, policy version, policy decision, evaluator result, human approval or rejection, output hash, model version, retrieval set and result status.

This turns the runtime state layer into a governance layer (as previously explored and mapped state layer where audit records accumulate) as audit records (including checkpoints, memory, trace IDs, policy decisions, and evaluator results) start to collect in an operational zone where said governance can either be searched evidence or become narrative archaeology.

C-Trace, a runtime compliance verification framework for AI agents, already looks at ways to transfer consent, purpose, data minimization and erasure as predicates over an execution trace for AI agent applications. The trace is generated by intercepting the tool invocations and model outputs of an AI application. Non-compliant actions are blocked. An audit log is created. Under perfect extraction, the system reports 0% attack success. Under 10% extractor noise, the system keeps attack success at or below 12%. The system keeps false positives at or below 16%. This is the result of an evaluation by the authors in the authors' evaluation.

The interesting part: the AI Act can be monitored in real time as the run happens.

Human oversight is a runtime event

Teams treat human oversight, and the related implementation into policy, as if it were only to do with the required staffing for implementing that policy. Name a human and assign a control.

Article 14 is more operational than the staffing slide. The Service Desk says oversight measures should enable assigned individuals to understand system capabilities and limitations, monitor operations, interpret outputs, override or reverse outputs, and interrupt the system by way of a stop button or similar procedure that brings the system to a safe state.

So that's an agent runtime. An approval is a state transition. An override is a recorded decision with actor, authority, input, output, and consequence. An interrupt is control flow.

This is where policy before the tool call comes in. Human oversight after the tool has already mutated a record is cleanup activity. Human oversight before risky activity is governance. The trace should clearly show this.

A lending agent can draft a credit recommendation. A human reviewer can approve, edit, or reject it. A healthcare triage agent can summarize patient history. A supervisor can stop an escalation. A support agent can prepare a deletion workflow. A privacy owner can verify scope before execution.

These decisions should also be captured as first-class events in the audit trail and not as after-the-fact comments in a case-management system.

Post-market monitoring is the part teams skip

The delayed high-risk dates will tempt executives to punt. Post-market monitoring has to become an operational habit before it becomes a compliance obligation.

Article 72 says providers of high-risk AI systems must establish and document a post-market monitoring system that actively and systematically collects, documents, and analyzes relevant performance data throughout the system lifetime to evaluate continuous compliance.

That phrase, throughout the system lifetime, carries the operational burden.

An agent changes after launch. A model changes after launch. A retrieval corpus changes after launch. A tool schema changes after launch. A permission boundary changes after launch (loosen up). A user cohort discovers a new path. The test cases never covered that new path. But one evaluator score means customer harm, the other score means no harm. The organization learns something new.

The important thing: post-market monitoring as a feedback loop. Trace samples from the field are used for evaluation. If the evaluation detects drift, then incidents are replayed and then used as test cases in release gates. The new runtime checks that are developed as part of the release gate process result in the agent getting safer and safer as more evidence is developed and flows in.

Runtime receipts are critical to financial workflows. A regulated workflow cannot be reduced to a summary of the overall performance of an agent. Rather, the review of such a workflow consists of a receipt containing a description of the action, the authority, the data, the approval, the outcome and the correction path.

What an audit packet contains

A full trace is too much for a reviewer to go through and a log line is too little detail to go by. A useful artifact for a reviewer is an audit packet cut from the trace.

For one agent action, that packet should contain:

Agent identity and runtime version.
Delegated user or service principal.
Session ID, trace ID, and parent action.
Tool, resource, and operation requested.
Purpose and data categories involved.
Policy version, policy decision, and reason code.
Evaluator scores or guardrail outcomes.
Human approval, edit, rejection, or interrupt event.
Output hash, result status, and downstream side effect.
Retention rule, tombstone status, and export location.

That sounds like work until the first concrete question arrives.

Note that the IETF Agent Audit Trail draft is an Internet-Draft and thus not a settled standard. However, for those interested in audit trail records from IETF software agents, this named the shape of JSON records containing the agent's identity, action taken, result of action, trust level, etc. that would form tamper-evident SHA-256 hash chains and optionally be ECDSA signed. JSONL, Syslog and CSV export are discussed as potential output formats for such records as a proposed audit format.

There's no need to wait for the standard to finish. The field names will change, but the architectural boundary will not. So, for example, if an agent is doing work that could affect people, money, access to information, employment, health, credit or regulated data, that work should create a record that can withstand a hostile read of that record.

The teams that start now get a boring advantage

The AI Act delay buys time for architecture, not time for governance-as-documentation theater.

The next year should be boring work. Firstly, put trace IDs on every run of the system. Secondly, start to version policy decisions. Thirdly, start to capture human interrupts as control-flow events, and then preserve evaluator scores. Finally, start to export action packets, write retention rules, and then above all else, keep raw sensitive data out of records unless a reviewer truly requires it to be in there. Rehearse the audit query before the audit exists.

This is the unglamorous side of ai agent governance. It will not demo well. Nobody claps for retention metadata.

But by then, the reason for the decision may have been forgotten. The team without runtime evidence has to interview the system after the fact.

AI Agent Testing Runs on Failure Traces | Focused Labs

Austin Vance — Mon, 27 Jul 2026 22:33:22 +0000

Agent tests should start from a failure under real traffic.

Teams want to re-run the exact failure in the trace to test the agent. A test case, not a screenshot, not a dashboard page, not a Slack thread of people agreeing to “watch it.”

That is where ai agent testing is heading. The useful unit of learning is no longer the benchmark score on Friday afternoon. It is the trace from a real run, turned into a verifier, owned by a team, and wired into the release path.

I don’t see the mysticalness fading much more from ai agent testing, but the interesting step in LangChain launching the Eval Engineering Skill is that the workflow now reads the repo, inspects the agent output, reads the traces, interviews the human who ran the task, and emits Harbor evals. The agent does not generate an opinion about quality. It generates a task the system can execute. The interesting part is that the agent is no longer only doing the work. It helps improve the engine that will run the next task.

I wrote this months ago: Traces collected from live agent behavior only become the basis for harness rewriting once an individual closes the loop from observed failure to change to the harness. This is fundamentally different from just trace storage, a pretty museum of weird agent behavior. Terrible for a test suite, nice for a touring show.

A score is a weak owner

Evaluations of agents have carried too much emotional weight. There is a simple number that a model score and a human score (a “judge” in the evaluation lingo) by definition get. Follow that up with a benchmark suite that scores by a few points here and there, and everyone pretends that that score corresponds in a simple manner to how an agent will actually behave under real traffic.

It does not.

Failures under real traffic have names: The agent picked the wrong tool. The retriever returned outdated policy text to the agent. Subagent mismeasured the error. Workflow failed to include required human in the loop. Agent attempted to write after a timeout and as a result duplicated side effect of the write.

An agent’s performance metric can improve in a benchmark setting while live traffic stays confusing. Conversely, a score does not say what broke. The trace says what is failing and why.

That distinction matters because ownership starts at the failure mode. LangChain’s IssueBench writeup, for instance, describes Engine processing batches of traces, then separating out the clean runs from the issue runs, and failing each of those into categories and attaching relevant traces to open issues or creating new issue cards for newly detected failures. In this way, the primary function of Engine’s input (traces) is turned into a usable stream of engineering work.

That is a healthier direction for agentic engineering. The artifact has to route. A hallucination, a silent tool error, a missing feature, and a bad recovery path belong to different owners. If the eval cannot preserve this, the team will be left with noise instead of work.

In Agent Failures Should Open Tickets, we wrote the ticketing version of this rule: recurring live-system failures ought to become owned work. In other words, trace + hypothesis + regression check. The new wrinkle is that the ticket should point at an executable eval opened by the failure.

The eval is a repo object

“eval” is still widely used to mean a spreadsheet (e.g. a balance sheet), a prompt (e.g. discussion prompt) a judge configuration (e.g. the evaluation configuration for a Harbor task). So all this feels pretty amorphous and soft.

A key distinction to keep in mind about Harbor artifacts is that they have boring form. The official Harbor task documentation describes a task directory with instruction.md, task.toml, a container environment, solution files, and tests. The core concepts documentation for Harbor says roughly the same thing: instruction, container environment, and test script.

An agent failure that matters should translate to something tangible: an instruction to reproduce the failure, the environment where it happened, the tools and constraints the agent had, a fixture or trace link, a verifier, and an owner who decides what change is allowed to make the test pass.

The trace matters when it becomes an executable artifact in the delivery loop.

This is where the primary ai agent workflow conversation needs to get concrete and this is the concrete part: the soft path (agent runs, agent answers, team observes dashboard in a web UI) vs. the useful path (live trace turns into failure cluster turned into an owned issue turned into a Harbor eval turn into a harness change turn into a rerun turned into a release gate).

Benchmarks should change the harness

The Deep Agents team is treating benchmarks similarly. In How We Benchmark Deep Agents, LangChain describes moving toward end-to-end Harbor evals for longer running tasks, then using those results to inform harness tweaks like removing todo-list middleware or slimming down the system prompt.

That is the right level of abstraction. The eval exists so evaluation steers the harness instead of decorating a model leaderboard.

This is where “ai agent workflow automation” gets tricky to use as a term. Automation conjures up images of more things running automatically. But the reality is to automate the feedback loop (agent fails, get trace, eval fails, fix on main (patches to prompts, tools, routing, context, middleware, model choice, etc.) and then the build proves the fix stayed fixed).

Deep Agents is a useful example because the harness is the product surface. A todo-list middleware seems benign until evals show that it distracts newer models or burns context. A longer system prompt looks safe until benchmarks show that it crowds out task instructions. A verifier that only checks the final answer looks fine until the trace shows that the agent skipped the required tool.

The reason to keep running evaluation after the release is that the data used for pre-release evaluation catches what the team had remembered to encode. And the traces from live evaluation catch what users, tools, permissions, and weird state transitions do to the plan after it has hit reality. Annoying. The job.

Graphs help when the loop has state

But for testing, graph-shaped agent systems make the story easier to follow. LangChain’s graph engineering essay describes LangGraph as a mix of deterministic paths and agentic steps, with nodes for work and edges for what happens next. The LangGraph Graph API shows conditional edges routing from graph state and Send fanning out work only known at runtime.

That matters because agent failures happen across transitions. The agent performed the first step correctly, then retried the wrong thing. It found the right document and sent it to the wrong specialist. It asked for approval while a sibling branch kept writing. It recovered from a tool error by losing the original intent.

A flat final-answer eval fails to capture that. A trace-linked task can assert the transition the agent took, and the verifier can check the produced artifact, issue category, approval state, absence of side effect, or route shape. That is closer to how agentic AI workflows break under real traffic.

An eval becomes useful when it has files, fixtures, a verifier, and an owner.

There are failures worth capturing in an eval artifact and failures that can wait. The rule of thumb is to capture failures that would cost customer trust, failures that recur from the same root cause, and failures that teams argue about after the fix. Everything else can probably stay in triage.

Approval is part of the workflow

GitHub’s new issue automation controls are a good adjacent signal to us. GitHub recently made a public preview available for agent automation in Issues. In this preview, the issue UI shows the rationale for label assignment, field assignment, issue type changes, closing, and assignment. Importantly, all of these approval panels, and associated automation, are merely workflow conveniences and do not map to any server-side security controls.

A release gate should test what the system enforces, not what the UI suggests. If an approval panel holds a label suggestion, test the suggestion path versus the agent directly applying the change. Test the permission boundary. Test the side-effect boundary when the workflow says rejection means zero side effects.

So, the release question is whether the trace, artifact, and verifier functioned properly under the same circumstances that previously failed.

This is why vibes are not an evaluation strategy. A person can like the demo. The dashboard looks great and the trend is up. The judge liked the final answer. The build still needs a particular failing case, and it must turn green when that failure mode is gone.

Own the loop

AI is simply another component in the larger engineering system. As Honeycomb’s CTO recently wrote in a letter to customers, AI amplifies the engineering system as it exists. Good practices get amplified. Bad practices get amplified. Engineering rigor matters more when AI enters delivery, not less.

A team with a healthy delivery system will turn traces into evals (because that is how the org already thinks, traces == evals). Failures create tests. Tests protect changes. Changes ship behind gates. Gates produce evidence.

A team without this muscle simply want a better dashboard. They’ll call out an interesting trace from time to time, rename a category here or there, add a sentence or two to a prompt. And of course the same failure will recur with a new name, dressed up in a new set of slides or a new Slack thread.

The engineering advantage is to treat every serious agent failure as a build opportunity. Mine the trace. Name the failure. Create the Harbor task. Assign an owner. Gate the release. Then use the release process to produce evidence that the fix held.

Then the agent workflow learns from real traffic instead of it being something it tries to survive.

AI Agents in Finance Live or Die by Runtime Receipts | Focused Labs

Austin Vance — Thu, 23 Jul 2026 22:13:45 +0000

The board wants to see the bottom line of the RFP. Show the receipt.

Financial services boards want to know what they get from the spend on AI in LangChain’s July post titled proving the ROI of agentic AI in financial services. In the case of agentic RFP responses, AML investigations, portfolio management workflows, and financial services compliance reviews, the work crosses documents, systems, approvals, policies, and people that are typically involved in such financial work.

One dashboard showing total tokens / uptime will not cut it.

Financial services already is stressful enough without questioning the value of AI. The data point: 68% of the financial-services respondents who used generative AI in a recent Snowflake survey from Snowflake were able to quantify a positive ROI from the technology, with 30% currently using agentic AI in live workflows. Moody's says Research Assistant users consume 60% more research and cut task completion time by 30%. As a result, financial services moves out of novelty category for agentic AI and goes straight to hard work of proving the value.

The hard part is proof.

Agentic AI ROI starts with work receipts

Agentic AI ROI quickly goes from clear to fuzzy as soon as teams start to measure the “wrong” unit. Spend is not value. Prompts are not work. A simple monthly usage graph proves activity. That’s to show a better board packet. (a pretty low bar).

The unit is the completed work item.

An RFP answer section is a work item. An AML alert disposition is a work item. A KYC exception package is a work item. A risk memo refresh is a work item with input data, a responsible agent, tool calls, model cost, human approvals, policy checks, writes to downstream objects, and so on, until a set of reviews culminates in an outcome for the item, the end result of the agent’s work. The end result has a set of receipts for the finance department (evals, traces, etc.). Each run of a work item through an agent’s tools generates one or more such end result receipts.

That is the receipt.

That connects to prior work on enterprise AI agents as runtime products. Finance needs the same owned boundary if it wants per-work-item proof.

LangChain’s finance piece names RFP processing and AML compliance monitoring as agentic use cases, then connects ROI to observability, governance, and economic measurement in the same stack. The operational discipline is making every meaningful run leave a receipt as it executes.

ROI proof is assembled while the agent runs, not after the fact in a spreadsheet.

The runtime is where the receipt gets written

Whatever live agents execute is already in full view of the runtime, so hiding all that inside engineering tools is the self-inflicted wound here.

An agent runtime handling real work knows the assistant, thread, run, input, tools, retries, costs, interrupts, stream events, and final state. LangSmith Deployment describes its Agent Server execution model around assistants, threads, and runs, with durable execution, real-time streaming, and horizontal scaling for agent workloads. The background-run API accepts metadata, context, webhooks, feedback keys, stream modes, and durability settings. That is enough surface area to attach business context before the work disappears into a trace search screen.

The receipt for a run should be created before the moment the run is created (i.e. before the first tool is called in that run’s execution).

Assign a task type to the run, attach the business object, cost center, policy, expected output contract, reviewer queue, and downstream system that the side effects will be written to. The runtime can then update the receipt as work is completed.

For AI agents in finance, the receipt becomes the control surface.

A receipt finance, risk, and engineering can read

A runtime receipt is boring on purpose.

That all gets recorded, and it is boring by design. There’s run ID and thread ID and task, data accessed, tool calls, model+token cost to run, latency to run, retries, approvals (and all the side-effect receipts), an evaluator score and a reviewer’s result, and business outcomes. For RFP sections we get to know if we’re accepting them with or without SME material re-write. For AML alerts they get escalated or closed or sent back for human review. For research memos they can be cited, rejected or re-written.

A receipt turns a trace into evidence finance, risk, and engineering can all read.

For Engineering it is debug evidence, for Risk it is authority and data lineage, for Finance it is cost per accepted work item, and for Product it is adoption tied to outcome. This receipt answers different questions for the teams and does not force each to make up its own “truth” for how well things are working.

Honeycomb has made similar observations in the past about Observability. As they put it in connect technical signals to business outcomes, because uptime, MTTR, and engineering productivity do not automatically yield a measure of what the business gets from the system. AI increases this already-substantial barrier, in that a healthy model, up service, etc. is not at all sufficient to guarantee that the agent using it will spend its time and money productively, or even at all.

AI agent cost is a runtime signal, and its useful question is whether the agent’s work item can be replayed, judged, explained, and tied to business results.

Aggregate dashboards are too late

They show spend by team, average latency, total runs, error rates, top agents, cost by model, and adoption trends. These are good to keep and to be proud of, but they are late by design, i.e. they cannot be used to prove the value of a single compliance-sensitive action.

Averages hide important details, especially when it comes to individual expensive runs and costly rework caused by high adoption. The expensive run might be valuable to investigate deeply. The cheap run might be garbage. The agent with high adoption might cause rework, whereas the agent with low adoption is the one doing the riskiest work for the firm.

The receipt is what lets the rollup make sense.

OpenTelemetry's graduation post is worth reading. Even though it’s talk of the future of telemetry and how to abstract it away from current vendor lock-in, it’s anchored firmly in current reality. OpenTelemetry is a graduated CNCF project and the maintainers point out that agent execution (running processes or containers) within agentic workflows (running workflows within workflow managers), and GenAI semantic conventions (as used by current AI and expected to be used in the future) are future workloads that OTel will be able to handle. Therefore, the receipt evidence that a run produced needs to be able to travel between all of the queues, databases, identity providers, SaaS APIs, and code review work-flows that the agent is executing in. The receipt cannot be locked in the private vocabulary of any single vendor.

Agent work volume turns measurement into architecture

Delegating work to agents as a regular part of one’s operating work is becoming operating volume, not a lab artifact. The recent Codex usage paper reported that active users grew more than fivefold in the first half of 2026. In addition, there are now more than 10% of users managing three or more agents in current active work at any time, and 26.6% of users using skills in their use of Codex for coding and other knowledge work.

Parallel execution breaks casual measurement.

The runtime needs to know what work item is currently active, which is pending approval, which handled customer data, which triggered follow-up jobs, which exceeded cost bounds, and which have side effects. A recent infinite-agent-loop paper found 68 confirmed failures across 47 projects. All of these failures can cause cost exhaustion, context growth, model denial of service, or repeated external side effects, and are caused by the iterations on the agent and its authority outpacing the system’s ability to observe its runtime behavior.

The Agent libOS paper gets the systems shape right: long-running agents require process identity, parent-child lineage, lifecycle state, explicit capabilities, human queues, checkpoints, events, and audit records. In financial-services language that becomes identity, authority, state, approval, evidence, and accountability.

Just as with money movement, spending authority moves into the runtime and therefore policy, approvals, receipts, and revocation need to be accounted for before the wallet signs the spending of tokens. The runtime should not simply spend tokens and hope finance reconstructs the value later.

Repeated work should become measured paths

The first version of the finance agent is an exploratory one. It will try out various tools, ask for more information here and there, fail evals (which it can recover from), request approval, and in the process learn which path is best.

Then the repeated paths should harden.

A weekly RFP process is not something that should stay in a loose agent loop forever. It’s a process of requirements extraction, approved-content retrieval, answers, citations, SME review, and packaging. And those should be a well defined process with well defined gates. (Even if there are also places for ambiguity).

That is the point behind agentic workflows getting less agentic. Repeated behavior of an agentic workflow should transform into an observable, doable, legible execution path (and remain agentic as soon as reality changes again). Likewise the ROI of a finance agent does not only grow as it becomes more agentic, it also grows as the agent’s work is first discovered, recorded, measured and then hardened.

And, side-effect layer. As we’ve written about before, side-effect receipts are critical to enabling features like retry, compensation, and ownership tracking when various tools are orchestrating work that causes side effects by running to update live systems. For a finance agent, this would include updating CRM records, requesting documents, opening up new case work, generating client-facing drafts, and opening up new compliance work, etc., all of which would need to be issued with an operation key and receive a corresponding receipt.

A trace in the workflow captures all the activity that occurred within the agent as it executed the workflow. A side-effect receipt, by contrast, captures the changes that the agent caused outside of the workflow, i.e., the actual side-effects.

Both belong in the ledger.

Own the ledger before the board asks

A distributed systems lecture is not what the board wants. Fair enough. They just want to know if their money has been converted into real work value.

To answer that question honestly requires a tremendous amount of engineering discipline to create runtime receipts that are dull enough to be understood by people in finance, risk, and engineering, and to attach all relevant metadata before work begins, to track all changes to authority throughout the process, to ensure that all side effects are recorded as first class facts, to tie the scores from evaluators and the results of human review to the work item in question, and to do rollups from the receipt as opposed to from the vibes that were brought to the work.

This also changes how one evaluates vendors for finance agent platforms. A platform without per-work receipts for finance, risk, and engineering to share is magic; a runtime with receipts is evidence, easier to defend when the model changes, when a workflow drifts, when a regulator asks for lineage, or when the CFO asks why the AI budget went up again.

Show the receipt.

Agent Protocols Fail at the Seams | Focused Labs

Austin Vance — Fri, 17 Jul 2026 21:45:42 +0000

The agent protocols are becoming integrated into normal software infrastructure, making integration easier. And this is making ownership blurrier.

These all make integration easier, so that’s great for product teams that want agents working together quickly. But then this becomes another failure surface in addition to specification. In this case, content flows quickly, and so do the different standards for delegation and state and all the rest. Meanwhile, ownership has traditionally moved more slowly through the org chart than that, so the mismatch becomes a problem for everybody.

Research is emerging around the integration of different agent protocols and the security of their composition. A 2026 paper, Formal Security Analysis of Agent Protocol Composition, investigates five agent protocols, finds 35 specification-level findings, backs them with 80 detailed implementation tests, and adds 30 failures found when the protocols were composed together as part of a larger system.

Finally, the years of pressuring AI agent vendors to connect together are starting to bear fruit. Work that can be done by isolated ‘agents’ is child’s play. The work of an enterprise is to cross boundaries in every direction. It has to traverse files, work items, schedules, databases, customer records in CRM systems, web pages viewed in browsers, software repositories, and APIs for internal services.

Clean locally. Sketchy globally.

Protocols standardize communication, not responsibility

Note to buyers: IBM’s useful definition says AI agent protocols standardize communication among AI agents and between agents and other system components; they do not standardize workflow coordination, execution, or optimization. Don’t make the stupid mistake of believing protocol support automatically includes control plane functionality.

MCP (Model Context Protocol) allows agents to discover tools and call them. A2A (Agent to Agent) allows independent agents to interact with each other using a standard interface. ACP (Agent Communication Protocol) allows agents to interact with each other using a REST interface (synchronous and asynchronous, streaming, stateful and stateless, online and offline, long running etc). It describes a thick grammar layer.

Grammar does not decide who can touch payroll.

The A2A 1.0 specification describes A2A in terms of three layers: the canonical data model, abstract operations, and bindings to specific protocols. Appendix A gives the enterprise pattern in miniature. An A2A client agent asks an A2A server agent to perform work. The server agent can then use tools and APIs supported by MCP. The architecture is useful for enterprise work and is exactly the direction people are moving AI agents.

ACP explicitly states the quiet part. The Agent Communication Protocol docs pitch interoperability between different frameworks, teams, and infrastructures. ACP exposes a REST API and is therefore opaque to the method calling the API. It supports long-running tasks, cancellation, streaming messages, and discovery. But as always: what looks valid as a method call can still be wrong when executed at runtime as part of a longer composed path of actions.

The risky boundary is where locally valid protocol calls become one composed path.

This is why MCP security starts after tool approval. Approving a server, registering its tools, and checking schemas only proves that a local component has a shape the client recognizes. Runtime security starts when the system asks a set of questions: What changed? What authority is being carried forward? What content entered the model’s context? What evidence will there be after the action is taken?

The failure moves through content, delegation, and authority

The AgentThread paper includes an example that is simple but should make security teams squint. One MCP server fetches attacker-controlled web content. Hidden instructions enter the model context. A second MCP server has file access. The agent reads local data and sends it back out through the first server. The paper describes the unsafe behavior as composition, implicit authority transfer, missing consent, and weak audit visibility across the agent runtime.

No single action needs to be out of the ordinary. Get a web page. Read a local file. Send the result of that.

The failure is the path.

The old framing of mcp security as a checklist around one server is too small. CoSAI’s Model Context Protocol Security whitepaper maps MCP to twelve threat categories and almost forty threats, including identity, access control, input and data boundaries, trust boundaries, lifecycle governance, logging, monitoring, and auditability. That taxonomy belongs in deployment design.

This is related to the prior post on the tool-call boundary. AI agent security happens at the tool call. But now that the tool call is a node in a larger composed route, the runtime must remember the content and the delegation that happened before calling that tool. A runtime that only checks the tool call is doing the bouncer’s work at one door, while there are tunnels in the building.

SDK conformance is not runtime safety

I like standards. I want fewer custom connectors. I want agents to connect to tools, agents, and servers without six months of mapping fields between corresponding data objects.

The AgentThread authors break down security failures at the protocol level to describe separate problems: unsafe behavior within a single protocol, missing SDK recommendations, missing deployment hardening, and unassigned responsibilities in cross-protocol cases. Ultimately, the authors find that agent-protocol insecurity stems from responsibility gaps across protocols, SDKs, and deployments, failures that occur when semantic content, delegated authority, and tool access cross protocol boundaries.

When a set of features required for securing agent-protocol interactions is distributed across three SDKs and documented in a wiki, no one ends up owning the seam.

Multi-agent systems already expose this kind of problem without the added complexity of security. We have written about how multi-agent systems break at the collaboration plane, at the claims, findings, handoffs, and shared-state level between agents. In the worst case, a lost finding turns into a security incident, and a lost authority boundary turns into a breach report.

Own the seam above the protocol stack

The fix starts by treating protocol composition as a runtime surface.

Standards committees can improve the grammar. Vendors can ship nicer SDKs. Security teams can review popular servers. Good. The runtime that joins MCP, A2A, ACP, AG-UI, browser tools, file tools, and internal APIs still has to own composition policy.

Standards define grammar. System ownership has to be assigned above them.

First, every step across protocol boundaries should leave an execution record: the principal, user or sponsor, original objective, delegated objective, source-content classification, requested tool or action, target resource, credential scope, policy decision, and resulting artifact. Call it a “ledger,” a “receipt chain,” or a “protocol-composition record.” The important thing is that the record exists for the entire composition path and is preserved for the duration of the execution.

As cross-protocol actions are executed, the authority granted to the previous step in the protocol should be restricted, such that subsequent steps in the protocol require additional justification or authorization, in line with the authority narrowdowns identified above. In particular, A2A delegation should not transfer the ambient authority of the calling principal to subsequent agents, MCP tool calls should be executed with credentials scoped to the task at hand as opposed to the human’s session, and browser-side agents should not silently inherit backend privileges as a result of a convenient protocol bridge.

Third, untrusted content needs to be labelled by the runtime as to its origin. This will allow policy to determine if later content can be used to justify actions that involve model processing of that untrusted content. A simple prompt-injection defense that only scans new text as it is injected into a model will not be effective if the injected text consists of, for example, web pages, ticket comments, email bodies, etc. Simply labelling such content as originating from the “user” will be insufficient to control its usage, since it would be allowed to be used to justify virtually any subsequent action.

Fourth, we need to carry trace context with us across protocol boundaries. As we said before, agent traces need to cross the MCP boundary because otherwise we would be left with planner traces with no tool/service spans, a useful story with a missing crime scene. But when protocol composition is involved, the downstream calls made by A2A or ACP can also cross MCP boundaries, and thus the traces they produce need to cross boundaries as well. If instead each of the different protocols produced a separate trace with no correlation between them, then any post-incident review would be pure archaeology.

Fifth, tracking of what to undo in case of failure (rollback) has to be attached to the composed path as well. The number of steps to undo can easily exceed the number of steps to complete a task already. If a tool creates a ticket, updates a file, sends out an email to customers, updates a CRM like Salesforce, and stores something in memory, then all of these steps have to be undoable as a whole. Otherwise the only thing to do in case of failure is to manually clean up the mess, which is what happens in enterprises already (classic).

The runtime is the security boundary buyers actually get

The protocol ecosystem is moving in the right direction. MCP lowered tool integration friction. A2A gives independent agents a shared way to delegate and report task state. ACP gives teams a simple HTTP path for agent interoperability. These are good tools for building useful workflows.

They are also incomplete by design.

A protocol specification describes messages, the state through which an agent goes, how an agent is authenticated, and the semantics of the operations that the agent can perform. But, the specific enterprise data structures, human rules for approval, scope of particular credentials, rules for handling an incident, and rules for rolling back actions that have been performed, all of these are outside of the scope of a protocol specification. They live in the runtime and application architecture around it.

So the procurement question should change.

Don’t just ask if the vendor supports MCP or A2A, because everybody supports the hot acronym eventually. Ask what happens when an agent delegates across A2A, calls an MCP tool, consumes untrusted content, accesses internal data, and returns an artifact to a user. Ask where the runtime’s ledger is. Ask how A2A-delegated credentials are minted for the task at hand. Ask how trace context is propagated throughout the composed protocol chain. Ask how policy sees the origin of new content introduced into a model. Ask how a system would roll back the damage done by a four-step process where each step seemed rational inside the bounds of the current protocol.

That is the AI agent security boundary now.

A protocol makes the path easier to build. The team running the system still owns what happens when authority crosses the seam.

AI Agent Infrastructure Is Splitting at the State Layer | Focused Labs

Austin Vance — Thu, 16 Jul 2026 21:40:53 +0000

The agent stack is so boring where buyers keep looking: the loop.

Loop in sight, the rest of the system is elsewhere in the loop (the agent loop did not disappear). Even more, a layer has been created to manage durable records of work done by the agent: state (a durable record of the work done by the AI agent to service customer requests). This is the crux of enterprise work. The enterprise will configure a system (AI or not) to create and maintain such records and also decide who can read such records, how long to keep them, what they are allowed to do with such records, etc. With the introduction of the state layer, AI agent infrastructure now splits at the state layer. Model + a prompt-centered agent no longer represents the infrastructure argument for AI agents.

The state layer of an AI agent is now the dividing line between the agent being a product and the agent being a parlor trick that happens to access Salesforce etc.

The loop is the wrong buying surface

The harder questions live one layer down.

Where does the state live after the tab has been closed? What writes the checkpoint? What happens when a tool call is successfully completed but the model fails before a summary can be generated? Where do traces, policy decisions and permission grants enter the run? What can an enterprise export when the vendor contract expires? Who can delete memory? Who can subpoena memory? Who is responsible for the record when two agents are working with the same customer and writing to the same file?

The durable boundary is where the agent becomes operating infrastructure.

State turns agent behavior into infrastructure.

The database is becoming the backend

On Thursday, LangChain and MongoDB announced a partnership that frames agent systems as requiring retrieval, persistent memory, operational data access, observability, and reliable deployment. The real launch is a set of tools, and the proof that the partnership matters sits in the “more about” line: "Agents need more than a model and a prompt". A tool-centered agent (or prompt-centered builder) is not a product.

That sentence is doing more work than the launch language around it.

The official LangGraph documentation notes that durable memory should use a database-backed checkpointer, with examples for Postgres, MongoDB, Redis, and Oracle. The LangSmith deployment documentation exposes the state-layer switch directly: set LS_DEFAULT_CHECKPOINTER_BACKEND to mongo and provide LS_MONGODB_URI. That line belongs in the platform review, right next to storage, deployment, and data residency.

Boring. Also the point.

State is where lock-in turns real

O'Reilly's 2026 AI agent architecture diagram defines six layers between the LLM and an AI agent running real workloads. Memory / persistent state is a first level primitive in the architecture above the vector database. This architecture matches the actual experience of the market for the framework layer: easy to switch out when the application is well bounded. State is much harder to switch out.

TGVP outlined a similar argument for how stateful services create moats because memory graphs, identity stores, policy logic, and audit history compound over time. While I dislike moat arguments for infrastructure (such as how custom UIs in enterprise software create a barrier to entry that replaces simply offering a “connector catalog” which can then be replaced by a single adapter), there is merit in the increased memory graphs, identity stores, policy logic, and audit histories that compound over time.

I wrote this out originally as part of my vendor evaluation criteria. The first part was model neutrality. This seems to be something people want to hear. Durable memory should use a database-backed checkpointer, and implementation details for that checkpointer should be exposed to review (storage, deployment, data residency, etc) long before the memory format, the trace (or log) schema, the policy and decision-making portions of the state, and the audit export. Open model routing on top of closed state is a nicer cage with better lighting.

Governance follows the state

The state layer also pulls governance out of slideware.

Agent governance can sound abstract and be confined to slideware when it is stored in policy documents. But as soon as the platform is viewed as a series of layers, the team starts to ask concrete questions about a given run of an agent. For example: who was the principal that started the run off? What was the task that narrowed down the set of permissions that were granted to that principal? What was the tool call that was made that actually crossed the boundary of what that principal was allowed to do with that set of permissions, given by that task? What was the policy that actually allowed that tool call to be made by that principal, given by that task? What was the checkpoint that actually recorded that run of the agent? And what was the trace that actually proves that that run of the agent happened in that order.

Research into the security of Agent-based systems is only just beginning to explore the boundary between what the agent is granted permission to do and what the MCP server will actually allow the agent to do. AgentBound looked at 296 popular MCP servers and found that automatically generated permission manifests worked without modification 80.9% of the time, with 0.6 ms average enforcement overhead. It is clear that MCP tool access is heading in the direction of explicit manifests and runtime enforcement of access rights rather than the current model of trust-by-default and running of host processes.

Identity work also dovetails nicely with this analysis. A 2026 AI identity report defines AI identity as a continuous relationship between what an agent declares and what it is observed to do. This relationship is established through the declaration, the observation, and the confidence in that observation. Thus the same architecture that tracks the state of a productive AI also establishes and tracks the identity of that AI. Declaration without observation is a credential. Observation without a durable store of state is a log. Confidence in the system’s understanding of the relationship between an agent’s declared identity and its observed behavior requires that the system have a durable store of state and be able to compare what has happened to what the system expected to happen.

Task-scoped access control, a topic we’ve covered in the past, also falls under this topic. A writeup for TrueFoundry’s TBAC (Task-Based Access Control) function explained how current identity-centric access models can strain under the weight of agents, because each task may need to call for a different permission set. That can happen quickly, and is particularly exposed to prompt-injection attacks. In short, TrueFoundry argues that task-based controls bundle minimal permissions for the duration of work. And that’s a great architecture, even if the vendor packaging leaves a bit to be desired.

We have looked at the authorization side before through tool-call security and agent principals (e.g. running an agent as a user). From a state-layer perspective, these all tie together: authorization, identity, memory, checkpoints, traces and audit records. The record of an agent doing work.

The stateful loop is the product

A live agent run has a rhythm.

Load memory. Call tool(s) and enforce policy for each. Write trace for each tool call. Record audit event. Update run state. Later, resume run from evidence collected during run. As mentioned previously, each live agent run will have a history which can be inspected by the platform. Such a history provides much greater value than having the agent’s runs be driven by prompts to which the model may or may not answer correctly.

State compounds.

The work does not have to be “unlearned” when an agent run is abandoned for whatever reason. A new run of work can start from the exact point at which the previous run left off.

However, all of this work would be for naught if the evidence from past runs of an agent were not properly monitored. This is why agent monitoring is infrastructure. The stateful agent loop that we discussed earlier includes a section for “write trace and audit records” for a reason. This is the means by which all of the side effects of a run of an agent are recorded, so that they can be inspected by a human (or automated system) later.

It is therefore no surprise that agent platforms that can handle real workloads are focused on platform work (as illustrated before).

The model may still be the expensive line item. The state layer is where operational trust accumulates.

Platform evaluation moves to the state layer

The buying checklist should change.

Where does it store the checkpoints for previous runs? What is the format of the platform’s memory? How does it join together the information from thread ID, run ID, trace ID, and audit ID? Does it record permission decisions made by a tool call with the record of that tool call, or does it have to rebuild that information from elsewhere? Can it export the state information on the platform surfaces, or is that something that one has to ask support to do for them? (Note that just because something is exported as a state, it doesn’t mean that the state information exported is the same as the information that was stored as state information on the platform surfaces). Can the system delete the memory of one user without affecting the memory of other users that are stored in the same index? How does the system partition out the information that it stores for state information by tenants, by environment, by region, by retention rules, etc.

Then ask the uncomfortable version: what gets lost during migration?

Prompt history and agent builder configuration are one thing. Checkpoints, memory, traces, approvals, audit records, and policy history are another. If the vendor can’t migrate all of that to the customer's new backend then the vendor is in control of the live behavior of the agent. Yes, the agent runs on the customer's cloud. Yes, the customer pays for the model and has a model key that the customer uses to sign into the agent. But the memory, the record of the agent’s governance, the record of the agent’s work, that lives somewhere else.

Own that layer or the agent's memory, governance and operating history belong to someone else's backend.

Agentic Workflows Should Get Less Agentic | Focused Labs

Austin Vance — Wed, 15 Jul 2026 03:32:07 +0000

Agentic workflows are supposed to get boring.

The first pass can be exploratory and pretty expensive. The tenth pass should be boring and not have to rediscover the solution through a completely new set of tools. That workflow has earned a promotion.

That is the useful part of the new Progressive Crystallization paper. It names a lifecycle agent teams are going to rediscover the painful way: agents explore, traces prove repeated behavior, tests turn that behavior into workflow code, and telemetry from those runs decides when the workflow gets demoted back to the agent layer.

The paper presents data from a cloud-network operations system handling real incidents in which deterministic executions went from 0% to 45% over eight months. In the meantime, the per-incident agent cost went down by more than 70% and the number of incidents doubled. The model did not simply get cheaper. Solved work stopped being agent work.

Exploration belongs upstream

We continue to see agentic workflows get treated as a fixed architecture choice: build an agent, give it tools, add approvals, add observability, and hope the loop gets better.

The better shape is a lifecycle.

The Progressive Crystallization taxonomy organizes the various possible forms execution can take in a lifecycle agent. The Type 3 form of execution is 'agent-orchestrated'. That is to say that an agent investigates a problem with bounded autonomy, checkpoints reads and routes writes through human approval. Type 2 forms of execution are 'hybrid'. In a hybrid execution a workflow of steps is owned by a workflow, but the model is used to interpret or classify information at scoped points in the workflow. Type 1 forms of execution are 'deterministic'. They consist of typed API calls, conditionals, and so on, that together form a rigid and non-negotiable execution path. There is no place in a Type 1 workflow for model invocation at runtime.

The maturity curve moves agency out of repeated execution and leaves it for novelty.

This is how enterprise teams build out agentic workflows. First, a novel incident gets explored with tools and state. Then, as that incident repeats, the team wants a workflow that implements that path of work. Then, as that workflow starts to get weird on the edge cases, the system needs a path back to exploration. And that's what we've been saying all along about LangGraph as the foundation for enterprise agents.

The compile step is the economic boundary

First we "reason" once through a problem. Then we execute that same problem solution over and over again as repeated incidents of the same thing. The Progressive Crystallization paper documents a real-world crystallization workflow, which turns out to be a straightforward incarnation of this fundamental software pattern.

The Compiled AI paper describes a class of live AI systems in which an LLM is used once to generate an artifact, that artifact is then validated, and then a large amount of code is run with zero execution tokens. The authors evaluate function-calling, reporting 96% task completion with zero execution tokens, and a break-even point of 17 transactions to compile the function. They report 57x reduction in tokens at 1,000 transactions.

If a workflow gets run repeatedly (i.e., it is a repeatable process), the fact that it is an agentic workflow gets to be a tax on the number of times that workflow gets run. That tax is enforced by runtime inference and, thus, is best countered by reducing the number of runtime inference steps, in particular the number of execution tokens required to run each step of the workflow. There are different approaches to do this (for example, model routing, prompt caching, reducing the size of the LLM used to implement each step of the workflow), all pointing at the same idea: a process repeated enough times becomes worth compiling into a function that can be called directly. We previously laid out the cost argument in AI Agent Cost Is a Runtime Signal, but that was at the level of an entire agent, whereas this tax can be incurred by individual workflows within an agent. That is the basic move behind Compiled AI.

Note, a workflow that is promoted will have a slightly different contract on the trace. The trace will serve as a candidate specification, which in the end would have to pass acceptance tests. The steps in the promoted workflow would have to have side effects that are idempotent (i.e. with keys, upserts, read-before-writes). The human approvals in the trace would have to be encoded in the workflow as explicit states instead of as mere chat turns. The failure modes of the steps in a promoted workflow would have to be of a particular type (i.e. typed exits).

LangGraph keeps the seam visible

So where does the seam go between exploration and execution?

LangGraph is especially useful in this situation because it does not force a team to decide between using a free-form agent loop, and building a full-fledged workflow engine. Instead, it lets a team use existing Python control flow and wrap API calls and other execution logic with persistence primitives, such as memory and checkpointing. Two primitives in the Functional API, @entrypoint and @task, are plain, and they end up powerful. An entrypoint defines a workflow, while a task defines work within a given workflow.

This plainness matters for two reasons. The first is that repeated execution of a particular path through the agent is just repeated execution of a function. Therefore that function can be written as Python code, complete with a model call in the middle if that still makes sense for the application. Second, the task API lets a team write one task for database records and another task for approval. In the first task, the write can carry an idempotency key, and the approval can carry its own. This way, the workflow can still do approval steps, and the workflow still gets the benefit of persisted runtime state for the agent, without doing that in a single, tricky prompt.

LangGraph also tracks changes to past workflow runs, and implements important implications for a reproducible workflow API. Saved workflow checkpoints are loaded into a resumed workflow from the beginning of the entrypoint, re-running @entrypoint calls and loading saved results from @task calls completed before the last checkpoint. The backward compatibility docs make the implication blunt: adding, removing, or reordering task calls before the last resume point can replay incorrect cached values. Nondeterministic code outside a task can also change behavior during replay.

Just like compiled code, a crystallized workflow is something that can be run again and again. It can have a version number. It can have a window of time in which to complete (a "drain window"). It can have a set of rules for migrating to a newer version in the middle of a run (a "migration plan"). And a path that has been promoted out of an agent loop is now something that behaves like API calls against a persisted record of the execution of that path. So it can be edited as a whole as a single "implementation", as a single program, rather than as a series of individual edits to individual prompts.

So the pattern here for live agentic systems, as outlined in the LangChain documentation for agent frameworks is to define a system of a combination of workflows and agents and let workflows be the simplest, cheapest, fastest and reliable means to complete a task when it doesn't require agency. Thus the mix of workflows and agents for a system will change over time.

Promotion is an evidence gate

The fastest way to ruin this workflow is to promote it based on vibes.

A model claiming a repeatable path is largely meaningless. A corpus of traces for same cases going through same path is significantly more valuable. A proposed workflow which passes tests for candidate specification is even more valuable. A promoted path of work with no human overrides, bounded by appropriate tool permissions, idempotent for all side effects, and clean rollback semantics, is starting to look like durable software.

First, approval queues belong in the runtime for agentic AI workflows. Once the path is known, agent orchestration belongs in code. Evals should sit close to where changes happen, and traces should sit close to the decision. So, the gate to "promotion" (i.e. to live usage of a particular path for a given case) is now just another release gate for a service.

The checklist of persistent state, re-executable work, idempotent writes, etc. to determine safe replay of the node is boring but there. That's what needs to be there for repeated agent paths to be 'promoted' to newly 'durable' function calls. Until then, they remain fragile in-the-loop workarounds with no persistent contract and hence no way to track, test or reproduce them for regression testing.

Observability decides demotion

Promotion gets the money conversation. Demotion keeps the system honest.

A deterministic workflow can decay. The API that a task uses can change. Firmware can be updated to add a field. A support policy can change. Workflows can continue to pass syntactic validation while slowly drifting away from the real work being done. The promoted system needs a way to notice this kind of decay without incurring the cost of the agent on every run. Traces become the control plane.

OpenTelemetry's GenAI observability post describes a single trace for an end-to-end workflow of invoking an agent, having a chat with a model, and executing a tool. Honeycomb's fast AI feedback loop writeup shows how similar telemetry can be used to drive SLOs, alerts, and dashboards, and can even be queried with Honeycomb's native query language.

This is the same reason Agent Traces Rewrite the Harness matters: traces of agentic workflow execution under real traffic matter. They become eval cases, they become harness patches, they become release gates, and they provide solid proof that fixes to solved problems actually stay fixed.

Observability turns agent traces into promotion evidence and drift into a demotion trigger.

The number of repeated incidents where the platform is still burning full agent loops, even after it has seen the pattern before, belongs in front of leadership as the waste signal.

Spend agency on novelty

The goal is not to remove agents from agentic workflows, that would be counterproductive and as silly as keeping all solved paths inside an agent loop forever.

Agents are valuable because there is always going to be uncertainty and changing reality: new requests, new incidents, misbehaving APIs, and policy exceptions between systems. The agent does all of its work at the boundary of the known process.

The operating mistake is letting that boundary stay frozen.

Workflows in live systems should decrease in agency over time. They spend fewer tokens to execute solved work. They save receipts for the execution of risky work. And they use the model for the part of the workflow where judgment still matters.

The Type 1:2:3 mix is the scoreboard. If every execution stayed Type 3 forever then the platform would be charging the business for amnesia.