DEV Community: Rashid Mahmood

The Anatomy of an Agent That Actually Works

Rashid Mahmood — Tue, 03 Nov 2026 05:01:29 +0000

Day 2 of 7 — The Agentic Engineer’s Field Notes

Yesterday I argued that the system is the artifact, not the prompt. Today I want to open the system up and show you what’s inside.

There is a diagram I draw on whiteboards more often than any other. It has four boxes and one arrow that loops back on itself. If your team can name what goes in each box and explain why the arrow loops, you have an agent. If they cannot, you have a chatbot pretending to be one.

Here it is, in plain text:

          ┌──────────┐
          │  MEMORY  │
          └────┬─────┘
               │
               ▼
   ┌───────────────────────┐
   │        MODEL          │◄────────┐
   └───────────┬───────────┘         │
               │ chooses             │
               ▼                     │
          ┌─────────┐                │
          │  TOOLS  │                │
          └────┬────┘                │
               │ produces            │
               ▼                     │
          ┌─────────┐                │
          │ RESULT  │────────────────┘
          └─────────┘
              the loop

Four moving parts. One loop. That is the entire substrate of agentic computing, and almost everything interesting in this field is about getting one of these four parts right.

Let’s take them one at a time.

1. The model

The model is the brain. It is also the part you have the least control over.

You did not train it. You cannot easily fine-tune it. It will be replaced under you, sometimes silently, sometimes with weeks of notice. Its behavior on your specific task is an empirical question, not a design choice.

This sounds like bad news. It is actually the most freeing thing about agentic engineering: the model is a capability, not an implementation. You don’t optimize the model itself. You optimize how you compose around it.

The practical consequence: choose the smallest, fastest, cheapest model that gets the job done, and design the system so that you can swap models without rewriting it. The teams that bet hard on a specific model’s quirks (“Claude does this best because of X”) look brilliant for six months and stranded for the next six. The teams that treat the model as an interchangeable component compound their improvements every time the frontier moves.

A useful heuristic that has hardened into a production pattern by 2026: route by difficulty. The dominant Claude Agent SDK pattern in production today uses Opus as the orchestrator and Haiku inside sub-agents. Cognition’s most recent writeup makes the same point with cross-frontier routing — running Claude and GPT together as “a capability router rather than a difficulty escalator,” each handling subtasks where it actually excels. Use a small model for routing, classification, and summarization. Reserve the frontier model for the actual reasoning. You will pay a fraction of the cost and most of your users will not notice.

2. The tools

The tools are the hands. They are also where most agents quietly fail.

A tool is a function the model can call. The model picks the tool, picks the arguments, sees the result, and decides what to do next. From the model’s point of view, your codebase is a vocabulary — every tool is a word it can speak. If your vocabulary is bad, no amount of model intelligence will save you.

I’m going to spend an entire day on tool design later this week, so I’ll keep it short here. The key insight is one Anthropic published explicitly in their tool-design guidance for engineers: more tools don’t always lead to better outcomes. Build a few thoughtful tools targeting specific high-impact workflows, not a thin wrapper around every API endpoint you happen to have.

A human-facing API and a tool designed for an LLM are not the same thing. Hold that thought until Day 4.

3. The memory

The memory is what survives across turns.

This is more subtle than it sounds. There are at least three kinds of memory in any non-trivial agent, and confusing them is the root of an entire class of bugs.

Working memory — the conversation so far. Everything the model can see in its current context window. This is bounded, and managing it is most of the game.
Episodic memory — past conversations and runs. “Last Tuesday the user asked us to summarize a contract. Here is what we did.” Stored externally; pulled in on demand.
Semantic memory — durable facts about the world or the user. “This user prefers SQL examples in PostgreSQL syntax.” Stored externally; sometimes pinned, sometimes retrieved.

Most “memory features” you’ve seen in chatbots conflate these. They store everything in a vector database, retrieve a top-k slice, and call it memory. That works for trivial cases and breaks the moment the user’s “memory” includes contradictions, retractions, or evolving preferences. We’ll get to the fix on Day 5, including the Letta team’s hierarchical memory architecture that has become the de facto reference design.

For now, the operating principle: memory is not a bag of facts. It is a managed surface that the agent reasons over. The management — what to write, what to retrieve, what to expire — is engineering, not magic.

4. The loop

The loop is what makes it an agent and not a function call.

In its simplest form, the loop is a while:

context = system_prompt + user_request
while not done:
action = model(context)
if action == "stop":
return action.final_message
result = execute_tool(action.tool, action.args)
context = update(context, action, result)

That’s it. That is the entire control flow of an agent. Everything else is decoration.

The loop is also where almost every interesting failure mode lives:

Infinite loops — the model keeps calling the same tool because it doesn’t realize the result is the same.
Off-track loops — the model wanders, retrying, exploring, never converging on the goal.
Dead-end loops — the model gives up too early because the context contains a misleading error.
Cost blowouts — the loop runs 47 iterations on a task that should have taken three.
Context exhaustion — the loop runs long enough that the original instruction has rolled out of the model’s attention window.

Every one of these is an engineering problem, not a model problem. You add max-iteration caps. You add cycle detection. You add cost budgets (the Claude Agent SDK now ships with a max_tool_calls budget defaulting to 12, for good reason). You add periodic context summarization. You add escape hatches that hand control back to a human. The loop is the place where agentic engineering becomes most like classical systems engineering — and most teams do not give it the attention it deserves because it does not feel like AI work.

Five patterns to know by name

When Anthropic published Building Effective Agents, they did the field a quiet favor by codifying five composable patterns. They are not new individually. What’s useful is that they are now named, with shared vocabulary across teams. If you are sketching an agent on a whiteboard, you should be able to point at any region of it and say “this part is evaluator-optimizer” or “this is parallelization” — and have the room nod.

The five:

Prompt chaining — sequential decomposition. Each step’s output feeds the next. Use when the task has obvious sub-steps and you want to trade latency for accuracy. (Generate marketing copy → translate it.)
Routing — classify the input, dispatch to a specialist. Use when categories of input genuinely need different handling, or when you’re routing for cost (small model triages, big model only when the triage flags hard).
Parallelization — run multiple model calls at once, either on independent sections or as a vote on the same question. Use for speed gains or for confidence-via-redundancy.
Orchestrator-workers — a central agent breaks the problem into sub-tasks dynamically and delegates. Use when the sub-task structure depends on the input and can’t be hard-coded. (This is the territory of multi-agent debates we’ll cover on Day 7.)
Evaluator-optimizer — one model proposes; another critiques; loop until acceptable. Use when there’s a clear quality criterion and iterative refinement measurably helps. (The Anthropic engineering team has explicitly used this for translation quality.)

Most production agents are some composition of two or three of these patterns. The error most teams make is reaching for the most complicated pattern they’ve heard of, when the simpler one would have shipped in a week.

The principle Anthropic states explicitly is the one to internalize: start simple. Add complexity only when simpler solutions demonstrably underperform.

A note on frameworks

You are going to be told, repeatedly, that you need a framework. LangChain, LangGraph, AutoGen, CrewAI, the Claude Agent SDK, the OpenAI Agents SDK, Letta, and a dozen others are all competing for your import statements.

Frameworks are not free. Each one introduces its own abstractions, its own coupling, its own state machine you have to understand to debug. For simple agents, a 50-line while loop around your model SDK is shorter, clearer, and easier to debug than the equivalent in any framework. For complex agents — multi-step, branching, retry-aware, observability-rich — a good framework will save you months. The trick is knowing where the cutover is for your team.

My rule of thumb: if you can describe your agent’s control flow in one paragraph of plain English, write it in raw SDK calls. If you cannot — if it has parallel sub-tasks, conditional branches, retries with backoff, persistent state across runs — pick a framework and commit. Don’t half-adopt one. The worst codebase I’ve debugged was a “lightweight wrapper” some team had built around three different agent frameworks at once.

The other rule: your framework should not be the most interesting part of your agent. If your team meetings are mostly about which framework to use, you are not building an agent. You are building an opinion.

The simplest agent worth shipping

Here is what a minimum-viable agent looks like in production. Not in a tutorial — in production.

A model, called via an SDK, with a system prompt that defines the role and the boundaries.
A small set of tools (often three to seven) that are designed for the model, instrumented with logging, and tested for the messy cases.
A memory layer that includes at least: a scratchpad for the current task, a way to recall recent runs, and durable preferences for the user.
A loop with a hard iteration cap, a cost cap, a wallclock cap, and an escape hatch that returns control to a human or to a deterministic fallback.
Observability hooks on every model call and every tool call, so that when something goes wrong (and it will), you can replay the trajectory.
One of the five Anthropic patterns as your top-level shape — usually orchestrator-workers or evaluator-optimizer for non-trivial tasks.

That’s the whole thing. You can build the first version in a week. The next six months are about making each of the four subsystems better — and that is what this series is about.

Tomorrow: the part that’s quietly eating prompt engineering

Look at the loop diagram again. Notice the arrow back into the model. Every time it fires, the context gets bigger. The model sees more, but its attention is finite. What it pays attention to determines what it does next.

The discipline of deciding what gets into the model’s context, in what order, with what compression — that is the most important emerging skill in this entire field. It has a name now. Karpathy gave it the canonical formulation last year: “the delicate art and science of filling the context window with just the right information for the next step.” It has a four-strategy framework that’s quietly become the reference model. And the teams that have internalized it are quietly running away from everyone else.

Tomorrow we go deep on context engineering. See you then.

Day 2 of 7. Yesterday: the mental shift. Tomorrow: context engineering — and why prompt engineering is dying as a job description. Follow along.

Sources & further reading:

Anthropic, Building Effective AI Agents
Anthropic, Writing Effective Tools for AI Agents
Cognition, Multi-Agents: What’s Actually Working
Claude Agent SDK overview

First published at codewithrashid.com.

Why Most Agents Never Survive Production (And the Mental Shift That Fixes It)

Rashid Mahmood — Sun, 01 Nov 2026 05:01:35 +0000

Day 1 of 7 — The Agentic Engineer’s Field Notes

The demo always works.

Picture it. A small audience, a slide deck, a prompt typed into a textbox. The agent reads a Jira ticket, opens a browser, scrapes a competitor’s pricing page, drafts a slide, posts it to Slack. Applause. Calendar invites for follow-up calls. Someone in product asks how soon it can ship.

Six months later, that agent is still not in production. Maybe it never will be.

In May 2026, this is no longer just an anecdote. Gartner is now publicly forecasting that over 40% of agentic AI projects will be cancelled by 2027. Industry surveys of shipped agent projects show a brutal split: narrow-scope projects (single workflow, defined boundaries) deliver on time at roughly 65%, with a median slip of about two months. Broad-scope projects (multiple workflows, multiple integrations) deliver on time at 16%, with a median slip approaching ten months. The gap is not the model. The gap is everything around the model.

If you are building agents in 2026 and your demo-to-production ratio looks like everyone else’s, this series is for you. Over the next seven days I’m going to walk through the system I use to build agentic systems that actually ship, hold up under load, and survive the next model upgrade. Today is the mental shift that has to happen before any of the rest of it lands.

Agents are not chatbots with tools bolted on

The most expensive mistake I see is treating an agent as “a chatbot, but it can also call functions.” That framing is intuitive and almost entirely wrong.

A chatbot is a stateless transformation: text in, text out. The user is in the loop. They read the answer, decide if it’s any good, and either accept it or rephrase. The chatbot’s mistakes are transient. A wrong answer in turn three is forgotten by turn five.

An agent is a closed loop where the model is making decisions, taking actions, and feeding the consequences of those actions back into its own context. The user is no longer the corrector. The model itself is the corrector. Mistakes don’t evaporate — they compound. A bad tool call at step three poisons the context at step four, which produces a worse decision at step five, and by step nine the agent is confidently wrong in ways that would have been caught instantly by a human in the loop.

Andrej Karpathy, in his 2025 LLM year-in-review, put a useful frame on what’s happening. We are not building “smarter chatbots.” We are summoning what he calls ghosts — entities optimized differently than biological minds, with what he labels jagged intelligence: simultaneously polymath-genius and grade-schooler-confused. They live on your computer. They make decisions. And, in his words, “it’s not just a website you go to like Google, it’s a little spirit/ghost that ‘lives’ on your computer.”

When you bolt tools onto a chatbot, you are not building a chatbot with hands. You are giving a ghost a loaded weapon.

The demo-to-production cliff

Most agent demos work because the demo path has been implicitly hand-tuned. The prompt has been iterated on the demo input. The tools have been tested on the demo input. The expected trajectory is two or three steps. The data is clean. The user is forgiving. The cost doesn’t matter because nobody is counting tokens during a slide.

Production looks nothing like that. A year of production agent deployments has surfaced a now-canonical list of failure modes. They show up in conference talks, post-mortems, and academic workshops (ICML even ran a Failure Modes in Agentic AI workshop earlier this year). The recurring themes:

Silent failures. Systems appear healthy. Outputs look reasonable. The agent has been confidently wrong for six weeks and nobody noticed.
Black-box decisions. When something does go wrong, no one can explain why the agent did what it did, because nobody instrumented the trajectory.
Permission explosion. Agents accumulate access over time. By month four, the customer-support agent has read access to half your data warehouse and nobody remembers approving it.
Runaway execution. A loop that should have stopped at iteration five runs to fifty. The bill arrives. The CFO has questions.
Compounding errors. Each step of a long-running agent introduces small errors. By step forty, the cumulative drift is fatal.
Context degradation. Surveys put this at the top of the list — 65% of developers now cite missing or rotted context as the leading cause of poor AI code quality.
Assumption-filling. When the agent doesn’t know something, it doesn’t ask. It guesses. The guess looks plausible. The downstream code calls a method that doesn’t exist.
Self-review blindness. Ask the agent if its work is good and it will tell you yes.

Each of these is solvable. None of them gets solved if your team’s mental model is “make the LLM smarter.” The model isn’t your bottleneck. The system around the model is.

Stop optimizing the artifact you’re staring at

When teams hit the production wall, their instinct is almost always to keep tuning the prompt. They add another bullet point to the system message. They reword the instructions. They paste in three more examples. They beg the model not to hallucinate.

This is a category error. The prompt is not the artifact. The system is the artifact.

The system has at least four moving parts: the model, the tools, the memory, and the loop that ties them together. The prompt is, at most, a configuration file for one of those. If your agent is unreliable, the prompt is one of maybe a dozen places the bug could live, and probably not the most likely. I have spent days reworking system prompts on agents whose real bug was a tool that returned a 200-OK with an empty body when the real answer was “not found.” The model dutifully reasoned over an empty body and produced confident nonsense. No prompt was going to fix that.

Once you internalize that the system is the artifact, your day-to-day work changes. You stop arguing about the wording of a sentence in the system prompt. You start instrumenting the loop. You start logging every tool call and every model output. You start asking: when the agent fails, which subsystem failed? That question is answerable. “Why is the LLM dumb today?” is not.

The shift, in one sentence

Here is the framing that changed how I work, and the through-line for everything in this series:

You are not building a smarter chatbot. You are building software whose runtime behavior is decided by a probabilistic component, with all the engineering rigor that statement deserves.

Read it twice. Every word in there is doing work.

Software — not a magic box. It has interfaces, state, error modes, and tests. It can be debugged. It has a deployment story.
Whose runtime behavior is decided by a probabilistic component — the model’s outputs are not deterministic and never will be. Your system has to be robust to that, the way distributed systems are robust to network failure.
With all the engineering rigor that statement deserves — observability, evals, rollback strategies, capability boundaries, cost budgets, incident response. Boring, unsexy infrastructure. The same things that make any production system work.

The teams that ship agents successfully are the teams that treat agentic systems as a new substrate for software engineering, not as a clever chatbot trick. Their mental model is closer to “I’m building a service that internally consults a brilliant but unreliable contractor on every call” than “I’m building a magic AI that does the job.”

This shift is not optional. It is, increasingly, the difference between the 16% who ship the broad-scope projects and the 84% who don’t. Anthropic’s 2026 Agentic Coding Trends Report tracks the inflection in real time: software development is shifting from writing code to orchestrating agents that write code. One case study profiles a team at Rakuten that had an agent implement a complex feature across a 12.5-million-line codebase in a single seven-hour run. That is not “AI helps engineers code faster.” That is a different shape of engineering work entirely.

The engineers who learn to do that work well — who can move fluently between the four subsystems, design tools an LLM speaks, build memory that doesn’t rot, run evals that catch silent regressions — those are the engineers who will be enormously valuable for the rest of the decade.

What the next six days look like

If you accept the shift, the natural follow-up question is: what does the engineering rigor actually consist of? That is what the next six days are about.

Tomorrow (Day 2): I’ll dissect the anatomy of an agent that actually works in production — the four moving parts, why frameworks often hurt before they help, and the simplest agent worth shipping. I’ll also walk through Anthropic’s five composable patterns (prompt chaining, routing, parallelization, orchestrator-workers, evaluator-optimizer) and where each one actually fits.
Day 3: Context engineering. Why Karpathy is right that this has eaten prompt engineering, the four-strategy framework that’s emerging as the canonical model, and why teams that figured this out a year ago are pulling away from everyone else.
Day 4: Tools. The most undervalued lever in agent design. Plus the lethal trifecta — Simon Willison’s framework that has predicted four major production AI vulnerabilities in the first two weeks of 2026 alone.
Day 5: Memory. The hardest unsolved problem in agentic systems. We’ll get into hierarchical memory architectures (Letta’s three-tier model, the LLM-as-OS paradigm), and why most “memory features” you’ve seen are theater.
Day 6: Evals. Hamel Husain’s discipline, codified. Error analysis, binary-not-Likert judgment evals, and the single habit that separates the teams whose agents quietly improve every quarter from the teams whose agents silently rot.
Day 7: The multi-agent debate — and the plot twist. Cognition famously published Don’t Build Multi-Agents in 2025. They published a follow-up in 2026 that significantly updates that position. We’ll go through both, plus Anthropic’s Planner/Generator/Evaluator architecture and what production multi-agent systems actually look like now.

Each piece stands alone. Together, they’re the playbook I wish someone had handed me three years ago.

A small ask before you go

If you take one thing from today, take this: stop describing your agent project by its prompt. Start describing it by its system. The next time someone on your team says “the agent is broken, can we tweak the system prompt?”, ask them which of the four subsystems they think is failing and what the evidence is. If they can’t answer, you’ve found your real problem — and it’s not the prompt.

Tomorrow I’ll show you what the four subsystems are, how they interact, and the diagram that has saved me more debugging hours than any tool I’ve ever installed.

See you then.

This is Day 1 of a 7-day series on agentic development. If this resonated, follow along — the rest of the week goes deep on context engineering, tool design, memory, evals, and multi-agent systems. No fluff, no hype, just what’s working in production in 2026.

Sources & further reading:

Anthropic, Building Effective AI Agents
Anthropic, 2026 Agentic Coding Trends Report
Karpathy, 2025 LLM Year in Review
ICML 2026 Failure Modes in Agentic AI workshop

First published at codewithrashid.com.

One broken call in a batch: how seven agent frameworks handle it

Rashid Mahmood — Wed, 28 Oct 2026 04:01:34 +0000

When a model returns several tool calls in one turn and one of them is broken, the framework has to decide what goes into the next request. agentic-arena asks two questions of that request: did the successful call's result reach the model, and was the broken call reported at all?

What was measured

Three kinds of broken call were tested, each sent in a batch next to a valid call: an unknown tool name, malformed arguments, and a missing required argument.

	unknown tool	malformed args	missing required arg
`vanilla`, `pydantic_ai`, `microsoft_af`	both	both	both
`langgraph`	both	good only	both
`smolagents`	error only	good only	error only
`openai_agents`	raises	both	both
`google_adk`	raises	raises	both

"both" means the good result and the error both reached the model.

Three ways to get it wrong

The results show three failure modes, and they do different amounts of harm:

Silent partial. The broken call vanishes with no message of any kind, and the run answers from partial evidence without the model ever being told.
Successful sibling discarded. smolagents does report the error, but as a rewritten task rather than a turn in the transcript. The good result is dropped along with the history, so the model sends the identical batch again.
Raises. This one is loud. The repo prefers it of the three because nothing is silently wrong.

Batching changes an earlier result

On its own, a langgraph call with malformed arguments produces no tool message and the graph halts, so the item fails where you can see it. Batched with a call that succeeds, the graph carries on and the drop becomes invisible. The repo records the visible failure as the better outcome. So "batching changes nothing" turned out to be false. The rule the repo checks instead is that batching may rescue a fault but must never make a framework worse than it is when calls run one at a time.

The probe had its own bug

The first version of this measurement counted the word "observation" in the messages a framework returned. One corpus entry describes Tokyo Tower as a "communications and observation tower", so the probe read 3 for a batch of 2. The overall direction of the results held, but one claim did not: it had said the model is never told, and smolagents does tell the model. The probe now matches each outcome's own text, and a test checks that none of those markers appear in the corpus.

Caveats

All of this runs in mock mode. The model's turns are scripted and byte-identical for every framework, so any difference comes from the framework's own error handling. These are behaviour measurements, not answer-quality rankings. No finding here says any framework produces a better answer; that needs a live run.

Reproduce

python -m pytest tests/test_parallel_tool_calls.py -q

The full write-up is in docs/findings.md, section 2b, and in the field guide.

Eight broken tool calls: how six agent frameworks recover

Rashid Mahmood — Mon, 05 Oct 2026 04:16:23 +0000

Models send bad tool calls: malformed JSON, a tool name that does not exist, a missing argument. What happens next is decided by the agent framework, not the model. In agentic-arena I measure that directly.

What was measured

The resilience arena feeds eight scripted faults to every framework. The scripted model turns are byte-identical for all of them, so any difference in outcome is the framework's own error handling:

res-01 calculator called with malformed JSON arguments
res-02 a tool that does not exist
res-03 an unevaluable expression (the tool returns ERROR)
res-04 a required argument missing entirely
res-05 an extra argument the tool does not accept
res-06 search with an empty query
res-07 an expression the safe evaluator must refuse
res-08 arguments serialised as JSON null instead of an object

Results

framework	recovered	notes
hand-rolled baseline	8/8	2 LLM calls each
pydantic_ai	8/8
microsoft_af	8/8
langgraph	7/8	fails `res-01`, malformed tool arguments
openai_agents	7/8	raises on `res-02`, unknown tool name
google_adk	6/8	fails `res-01` and `res-02`, both uncaught exceptions
smolagents	8/8	but 6 LLM calls on four faults, against 2 elsewhere

The smolagents split is exact. On the four faults where the tool ran and returned something, it recovers in two calls like everyone else. On the four its validation layer rejects first (unknown name, missing argument, unexpected argument, null arguments), the error never reaches the transcript. The model cannot see it, re-emits the identical call, and the run spends the whole six-call budget, about 2.7x the prompt tokens, with zero tool calls recorded. It still reaches the right final answer, so the item passes. A retry wrapper would not help: the prompt is byte-identical every time.

Google ADK is the only framework that loses both res-01 and res-02. Both are uncaught exceptions rather than the model giving up, which is at least loud, and the kind of failure a retry wrapper can handle.

A correction

The smolagents row read 4/8 for several iterations. That was measured when an exhausted run returned an empty string; it now surfaces a final answer from its last memory step, so the check passes. The mechanism is unchanged, but the consequence is roughly 3x the cost, not a lost item. check_resilience.py now gates every count in this table, so the next drift fails CI instead of ageing silently.

Caveats

These are scripted-model (mock) behaviour measurements, not answer-quality rankings. Mock mode compares two things honestly: what a framework sends on the wire, and how it behaves when the model misbehaves. This arena is the second. It says nothing about how well any framework does with a real model on real tasks.

Reproduce

python -m arena run --arena resilience --framework all --mode mock --no-scorecard && python .github/scripts/check_resilience.py

Full findings, with the command behind every number: https://code-with-rashid.github.io/agentic-arena/findings/

What agent frameworks cost on the wire: measurements from agentic-arena

Rashid Mahmood — Wed, 30 Sep 2026 17:05:46 +0000

Most framework comparisons argue from feature lists. I wanted numbers, so agentic-arena holds the model, tools, datasets and iteration budget fixed and measures what each framework does differently.

Read this first: everything below is measured against a scripted (mock) model, so the turns are byte-identical for every framework. That makes these wire and behaviour measurements. They say nothing about which framework writes better answers.

1. What a framework puts on the wire

Mean prompt tokens per item on a 15-item tool-use task:

framework	prompt tokens	vs baseline
vanilla (stdlib loop)	753.5	1.00x
langgraph	753.5	1.00x
pydantic_ai	794.0	1.05x
microsoft_af	802.0	1.06x
google_adk	836.1	1.11x
openai_agents	856.9	1.14x
smolagents	2935.5	3.90x

Six of seven sit inside a 1.15x band, and none is leaner than the hand-rolled loop. smolagents' 3.90x is a 4,207-character system prompt where the arena asked for 384, including a prose restatement of tools it already sent as a schema. The tools are transmitted, and billed, twice.

That is a worst case. The same conversation run to 30 turns shrinks the gap from 8.83x on the first request to 1.27x on the 31st, because a fixed overhead decays as the conversation grows.

2. What breaks when the provider fails

I scripted 429, 500 and 400 responses. The hand-rolled baseline has no retry at all, so one 429 loses the item. Every framework survives a single 429. Only smolagents survives three in a row, by quietly sleeping roughly two to four minutes on one item. The item passes, so nothing in a scorecard shows it, but in a batch your throughput quietly collapses.

3. What delegation costs

A three-role pipeline doubles the LLM calls and multiplies prompt tokens by 2.5x, whether you build it with a graph library or a or loop. The structure costs that, not the framework.

Model-decided handoffs cost about 10% more prompt than the same pipeline wired structurally. Of that gap, 94% is the ransfer_to_* tool schemas, which ride on every request whether or not anyone delegates. You pay for the options you offer, not the ones you use.

The benchmark found bugs in itself

Five of the seven adapters never passed the shared request timeout to their client, so they waited out their library's default through a 20-second hang on a one-second budget. A hanging mock provider exposed it. It is fixed, and gated in CI.

Reproduce it

Everything above has a command that regenerates it, and CI regenerates each number on a clean install:

git clone https://github.com/code-with-rashid/agentic-arena
cd agentic-arena
python -m pip install -e .
python -m arena run --arena tool_use --framework all --mode mock --no-scorecard

Full findings and the reasoning behind each: https://code-with-rashid.github.io/agentic-arena/findings/