Source: This article summarizes the ~15-minute talk by Barry Zhang (Anthropic) posted by @Ronycoder on X. Presented at the "Agents at Work" event.
Introduction
Barry Zhang at Anthropic co-authored the blog post "Building Effective Agents" with Eric. In this talk, he goes deeper on three core ideas from that post and shares practical lessons on building AI agents in production.
Three core principles:
- Don't build agents for everything
- Keep it simple
- Think like your agent
The Evolution of AI Systems
Barry opened with a recap of how AI systems have evolved.
| Phase | Description |
|---|---|
| Single-LLM Features | Summarization, classification, extraction — one LLM call, simple and complete |
| Workflows | Multiple LLM calls orchestrated by code in predefined control flows. Allows trading off cost vs. agency for better performance |
| Agents | LLMs deciding their own trajectories, operating almost independently based on environment feedback |
| Next phase (?) | More general-purpose single agents, or multi-agent collaboration — still unknown |
The trend is clear: more agency means more capability and usefulness, but also higher cost, latency, and consequences of errors. This tension is the foundation for all three principles.
Principle 1: Don't Build Agents for Everything
Agents are a way to scale complex and valuable tasks — not a drop-in upgrade for every use case. So when should you build one? Barry presented a four-item checklist.
Check 1: Task Complexity
Agents thrive in ambiguous problem spaces. If you can map out the entire decision tree fairly easily, just build that explicitly and optimize each node. It's more cost-effective and gives you more control.
Check 2: Task Value
Agents consume a lot of tokens. If your budget per task is around 10 cents (e.g., a high-volume customer support system), that only affords 30–50 tool-using tokens — use a workflow instead. On the other hand, if your first thought is "I don't care how many tokens I spend, I just want to get the task done," agents are the right call.
Check 3: De-risk Critical Capabilities
Before running an agent, check for significant bottlenecks in its trajectory. For a coding agent: can it write good code, debug, and recover from errors? Gaps here aren't fatal but will multiply your cost and latency — reduce scope and try again.
Check 4: Cost of Error and Error Discovery
If errors are high-stakes and hard to discover, it's difficult to trust an agent with autonomous actions. You can mitigate with read-only access or human-in-the-loop, but these also limit how well the agent scales.
Why Coding Is a Great Agent Use Case
Barry walked through the checklist with coding as an example:
- Complexity: Going from design doc to PR is clearly ambiguous and complex
- Value: Good code is highly valuable
- Capability: Models are already great at many parts of the coding workflow
- Verifiability: Output is easily verified through unit tests and CI — this is probably why so many successful coding agents exist today
Principle 2: Keep It Simple
"Agents are models using tools in a loop."
Barry defined agents with this single sentence and identified three components.
env = Environment()
tools = Tools(env)
system_prompt = "Goals, constraints, and how to act"
while True:
action = llm.run(system_prompt + env.state)
env.state = tools.run(action)
| Component | Role |
|---|---|
| Environment | The system the agent operates in (filesystem, browser, APIs, etc.) |
| Tools | Interface for the agent to take actions and get feedback |
| System Prompt | Defines the agent's goals, constraints, and ideal behavior |
A coding agent, a search agent, and a Computer Use agent look completely different on the surface — but internally at Anthropic, they share almost the same code backbone. Only the environment varies by use case; the real design decisions are which tools to offer and what the system prompt says.
Complexity Kills Iteration Speed
The reason Barry emphasizes simplicity is empirical: any upfront complexity kills iteration speed. Build these three components first — the highest ROI comes from getting them right. Optimization comes later.
Examples of optimization after the basics are solid:
- Coding / Computer Use: Cache the directory to reduce cost
- Search (many tools): Parallelize tool calls to reduce latency
- All cases: Present the agent's progress to gain user trust
Principle 3: Think Like Your Agent
"Put yourself in their shoes context window"
Agents can exhibit remarkably sophisticated behavior. But at each step, what the model is actually doing is running inference on a limited context of 10–20K tokens. Everything the model knows about the current state of the world fits in that context window.
Barry illustrated this with a Computer Use exercise:
"Imagine you are a Computer Use agent."
You receive a static screenshot and a poorly-written description. The only way you can affect the environment is through your tools. When you attempt a click, while inference and tool execution are happening, it's equivalent to closing your eyes for 3–5 seconds and using the computer in the dark. Then your eyes open and you see another screenshot. Whatever you did could have worked — or you might have shut the computer down. You just don't know.
After going through this (mildly uncomfortable) exercise, what the agent actually needs becomes obvious:
- Screen resolution (so it knows where to click)
- Recommended actions and limitations (guardrails to avoid unnecessary exploration)
Using Claude to Evaluate Claude
Barry shared a practical technique: throw the entire agent trajectory into Claude and ask:
"Why do you think we made this decision here? Is there anything we could do to help you make better decisions?"
You can also pass your system prompt directly to the model and ask "Is anything ambiguous? Can you follow this?" This shouldn't replace your own understanding of the context — but it gives you a much closer perspective on how the agent sees the world.
Three Open Questions for the Future
Barry closed with three things that are always on his mind as an AI engineer.
1. Budget-aware agents
Unlike workflows, we don't have good control over cost and latency for agents. Defining and enforcing budgets in terms of time, money, and tokens would unlock many more production use cases.
2. Self-evolving tools
We're already using models to help iterate on tool descriptions. This should generalize into a meta-tool where agents design and improve their own tool ergonomics — making agents dramatically more general-purpose.
3. Multi-agent collaboration
Barry is personally convinced we'll see significantly more multi-agent collaboration in production by end of 2026. The benefits are clear: parallelization, separation of concerns, sub-agents protecting the main agent's context window. But the open question is: how do agents actually communicate with each other? Moving from synchronous user-assistant turns to asynchronous communication and mutual recognition between agents is the next frontier.
Summary
| Principle | Key Point |
|---|---|
| Don't build agents for everything | Evaluate use cases on 4 dimensions: complexity, value, capability, error cost |
| Keep it simple | Start with just Environment + Tools + System Prompt, optimize later |
| Think like your agent | Step into the context window and experience the world as your agent does |
Getting agents to work reliably is still hard — but much of that difficulty comes from developers not understanding what the agent sees and doesn't see. These three principles are a practical starting point for bridging that gap.
Full Transcript
The following is a translated and lightly edited transcript of the talk. Some recognition errors may be present.
[0:00–5:00] Part 1: The Evolution of Agents and the "Don't Build" Checklist
My name is Barry, and today we're going to be talking about how we build effective agents. About two months ago, Eric and I wrote a blog post called "Building Effective Agents." In there, we shared some opinionated takes on what an agent is and isn't, and we gave some practical learnings that we've gained along the way. Today, I'd like to go deeper on three core ideas from the blog post and provide you with some personal musings at the end.
Here are those ideas. First, don't build agents for everything. Second, keep it simple. And third, think like your agents.
Let's first start with a recap of how we got here. Most of us probably started building very simple features — things like summarization, classification, extraction. Really simple things that felt like magic two to three years ago and have now become table stakes. Then, as we got more sophisticated and as products matured, we got more creative. One model call often wasn't enough. So we started orchestrating multiple model calls in predefined control flows. This basically gave us a way to trade off cost and agency for better performance, and we called these workflows. We believe this is the beginning of agentic systems.
Now, models are even more capable. And we're seeing more and more domain-specific agents start to pop up in production. Unlike workflows, agents can decide their own trajectory and operate almost independently based on environment feedback. This is going to be our focus today.
It's probably a little bit too early to name what the next phase of agentic systems is going to look like, especially in production. Single agents could become a lot more general purpose and more capable, or we could start to see collaboration and delegation in multi-agent settings. Regardless, the broad trend is that as we give these systems more agency, they become more useful and more capable — but as a result, the cost, latency, and consequences of errors also go up.
That brings us to the first point. Don't build agents for everything. We think of agents as a way to scale complex and valuable tasks — they shouldn't be a drop-in upgrade for every use case.
We talked a lot about workflows in the blog post because we really like them. They're a great, concrete way to deliver value today. So when should you build an agent? Here's our checklist.
The first thing to consider is the complexity of your task. Agents really thrive in ambiguous problem spaces. If you can map out the entire decision tree pretty easily, just build that explicitly and optimize every node. It's a lot more cost-effective and gives you a lot more control.
Next is the value of your task. Agents will cost you a lot of tokens, so the task really needs to justify the cost. If your budget per task is around 10 cents — for example, you're building a high-volume customer support system — that only affords you 30 to 50 tool tokens. In that case, just use a workflow to solve the most common scenarios. You'll capture the majority of the value from there. On the other hand, if your first thought is "I don't care how many tokens I spend, I just want to get the task done" — please see me after the talk, the go-to-market team will love you.
From there, we want to de-risk the critical capabilities. Make sure there are no significant bottlenecks in the agent's trajectory. If you're doing a coding agent, make sure it's able to write good code, debug, and recover from errors. If you do have gaps, they probably won't be fatal, but they will multiply your cost and latency. In that case, reduce the scope and try again.
Finally, the last important thing to consider is the cost of error and error discovery. If your errors are going to be high-stakes and very hard to discover, it's going to be very difficult to trust the agent to take actions autonomously. You can mitigate this by limiting scope, using read-only access, or adding human-in-the-loop — but this will also limit how well you can scale.
Let's see this checklist in action. Why is coding a great agent use case? First, going from a design doc to a PR is obviously a very ambiguous and complex task. Second, we know good code has a lot of value. Third, many of us already use Claude for coding, so we know it's great at many parts of the coding workflow.
[5:00–10:00] Part 2: Keep It Simple — The Three-Component Design
And last, coding has this really nice property where the output is easily verifiable through unit tests and CI. That's probably why we're seeing so many creative and successful coding agents right now.
Once you find a good use case for agents, this is the second core idea — keep it as simple as possible. Let me show you what I mean. This is what agents look like to us. They are models using tools in a loop. In this frame, three components define what an agent really looks like.
First is the environment — the system that the agent is operating in. Then we have a set of tools, which offer an interface for the agent to take action and get feedback. Then we have the system prompt, which defines the goals, the constraints, and the ideal behavior for the agent to work in this environment. Then the model gets called in a loop — and that's agents.
We've learned the hard way to keep this simple, because any complexity upfront is really going to kill iteration speed. Focusing on just these three basic components is going to give you by far the highest ROI. Optimizations can come later.
There are three agent use cases that we've built for ourselves or our customers. They look very different on the product surface, very different in scope, very different in capability. But they share almost exactly the same backbone — almost the exact same code.
The environment largely depends on your use case. So really, the only two design decisions are: what are the tools you want to offer to the agent, and what is the prompt you want to instruct your agent to follow.
Once you've figured out these three basic components, you have a lot of optimization to do. For coding and computer use, you might want to cache the directory to reduce cost. For search, where you have a lot of tools, you can parallelize to reduce latency. And for almost all of these, we want to make sure to present the agent's progress in a way that gains user trust. But that's it. Keep it as simple as possible as you're iterating. Build these three components first, then optimize once the behavior is stable.
[10:00–14:46] Part 3: Think Like Your Agent & Open Questions
All right, this is the last idea — think like your agent. I've seen a lot of builders, myself included, who develop agents from their own perspective and get confused when agents make a mistake. It seems counterintuitive to us. That's why we always recommend getting into the agent's context window.
Agents can exhibit some really sophisticated behavior — they can look incredibly complex. But at each step, what the model is doing is just running inference on a very limited set of context. Everything the model knows about the current state of the world is explained in that 10 to 20K tokens. It's really helpful to limit ourselves to that context and see if it's actually sufficient and coherent. This gives you a much better understanding of how agents see the world and helps bridge the gap between our understanding and theirs.
Imagine for a second that we're Computer Use agents now. What we're going to get is a static screenshot and a very poorly written description. We have tools and a task. We can think and reason all we want, but the only thing that's going to take effect in the environment is our tools.
So we attempt a click without really seeing what's happening. While inference is running and tool execution is happening, this is basically equivalent to closing our eyes for three to five seconds and using the computer in the dark. Then we open our eyes and see another screenshot. Whatever we did could have worked — or we could have shut down the computer. We just don't know. It's a huge leap of faith, and the cycle starts again.
I highly recommend trying to do a full task from the agent's perspective. It's a fascinating and only mildly uncomfortable experience. But once you go through it, it becomes very clear what the agent would have actually needed. It's crucial to know the screen resolution. It's helpful to have recommended actions and limitations — guardrails to avoid unnecessary exploration.
Fortunately, we're building systems that speak our language. So we can just ask Claude to understand Claude. You can ask if any instructions in your system prompt are ambiguous. You can throw in a tool description and see whether the agent knows how to use the tool. And one thing we do quite frequently is throw the entire agent's trajectory into Claude and ask: "Why do you think we made this decision here? Is there anything we can do to help you make better decisions?"
This shouldn't replace your own understanding of the context, but it will help you gain a much closer perspective on how the agent is seeing the world.
All right, I've spent most of the talk on practical stuff. Let me indulge myself and share one slide of personal musings — my view on how this might evolve, and some open questions I think we need to answer together as AI engineers.
First, I think we need to make agents a lot more budget-aware. Unlike workflows, we don't have great control over cost and latency for agents. Figuring this out will enable many more use cases by giving us the control we need to deploy them in production. The open question is: what's the best way to define and enforce a budget in terms of time, money, and tokens?
Next is the concept of self-evolving tools. We're already using models to help iterate on tool descriptions. But this should generalize into a meta-tool where agents can design and improve their own tool ergonomics. This will make agents a lot more general-purpose as they adopt the tools they need for each use case.
Finally — and I don't think this is a hot take anymore — I have a personal conviction that we'll see a lot more multi-agent collaboration in production by end of this year. They're well-parallelized, they have nice separation of concerns, and having sub-agents will really protect the main agent's context window. But the big open question is: how do these agents actually communicate with each other? We're currently in this very rigid frame of mostly synchronous user-assistant turns. How do we expand from there and build asynchronous communication? Roles that afford agents to communicate and recognize each other? I think that's going to be a big open question as we explore this multi-agent future.
If you forget everything I said today, here are the three takeaways. First, don't build agents for everything. If you do find a good use case, keep it as simple as possible for as long as possible. And finally, as you iterate, try to think like your agent — gain their perspective and help them do their job.
Top comments (0)