DEV Community: Paramanantham Harrison

Workflow or agent: who decides what runs next

Paramanantham Harrison — Tue, 13 Oct 2026 19:20:46 +0000

When a team shows me an AI feature that is slow, expensive or unpredictable, the cause is rarely the model. It is a decision made in the first hour and never revisited: workflow or agent, and if agent, one or several.

Whoever decides what runs next, your code or the model, sets what the system costs, how it fails and how you debug it. That is the argument of this issue, and the evidence for it comes from the teams who build agents for a living.

If your code decides, you have a workflow. If the model decides, you have an agent. If one model hands work to other models, you have a multi-agent system. Each one is a different bill, a different kind of failure and a different way to debug.

How we know

We read the primary write-ups from teams that build agents for a living, Anthropic, Cognition and Sierra, plus a peer-reviewed study of multi-agent failures, and quote each number in the sentence its source states it in.

Make the workflow your default and let the agent earn its place

Anthropic draws the line in two sentences. Workflows are systems where models and tools are orchestrated through predefined code paths [1]. Agents are systems where the model directs its own process and tool use [1]. The line is not about how clever the model is; it is about who holds the steering wheel.

A support ticket, handled as a workflow

Classify the ticket: One model call labels it billing, bug or account
Route by the label: A plain switch picks the next step, not the model
Draft the reply: One model call writes it from the account's history
Approve and send: A person reads the draft before it goes out

Every arrow in that track is code you wrote. You can unit test it, retry a single step and explain a bad reply to an auditor. The model does real work inside two of the boxes, but it never decides which box comes next.

Anthropic's advice is to find the simplest solution possible and only increase complexity when needed [1]. Their observation from customer work is that the most successful implementations use simple, composable patterns rather than complex frameworks [1]. A framework that hides the loop makes it easy to hand the model a decision you never meant to give it.

Most business automation belongs here: triaging an invoice, summarising a ticket, drafting a reply for approval, enriching a lead. The path is known before the request arrives, so there is nothing for a model to decide about the order of steps. Give it the judgement inside a step, and keep the order in code.

Every decision you hand the model is paid for in tokens

Anthropic measured what its own research feature spends. Agents typically use about 4× more tokens than chat, and multi-agent systems about 15× more [2]. Each extra decision is another call that rereads the context so far.

Tokens used, compared with one chat exchange [2]

A chat exchange: baseline
A single agent: 4×
A multi-agent system: 15×

That spend buys something. In the same study, token usage by itself explained 80% of the variance in how well the system did [2]. More tokens meant more searching and better answers, on a research task where breadth is the whole point.

The trade is stated plainly: agentic systems often trade latency and cost for better task performance, and you should consider when this tradeoff makes sense [1]. For a ticket router that runs ten thousand times a day, it rarely does. For a research question an analyst asks once, it often does.

So price one request before you build. Count the model calls on the happy path, multiply by the context each one carries, and compare that with what the task is worth. If you cannot write that figure down, you are not ready to pick an architecture.

An agent that succeeds once has not shown it will succeed twice

Sierra built a benchmark of support conversations with simulated customers, real tools and real policies. All 12 agents they tested had difficulty, and even the best one stayed under 50% average success [3]. These were not toy tasks; they were the returns and booking changes a support team handles every day.

60% drop in the best agent's score when the same retail task had to succeed in eight tries instead of one. [3]

That drop is the number to worry about. The best agent fell to about 25% when the same retail task had to succeed on all eight tries [3]. A workflow that works on Monday works on Tuesday. An agent that works on Monday has rolled the dice once.

At Sierra, our experience in enabling real-world user-facing conversational agents has made one thing extremely clear: a robust measurement of agent performance and reliability is critical to their successful deployment.

Karthik Narasimhan, Head of research, Sierra [3]

Errors also compound. Anthropic notes that minor issues for traditional software can derail agents entirely, because each step builds on the state the last one left [2]. One wrong tool result early in a run becomes the premise for everything after it.

The fix is not a better prompt. It is a habit: log every trace from the first day, read them, and turn real failures into test cases. Hamel Husain, who has helped many teams ship products built on language models, names the cause of failure without hedging [4].

I’ve found that unsuccessful products almost always share a common root cause: a failure to create robust evaluation systems.

Hamel Husain, AI consultant and educator, Parlance Labs [4]

A second agent pays for itself only when the work splits cleanly

Multi-agent systems do win under the right conditions. Anthropic's multi-agent setup beat a single agent by 90.2% on its internal research eval [2]. Research splits well: many independent searches, each returning a short summary to a lead agent.

90.2% better than a single agent on Anthropic's internal research eval, where subagents search in parallel. [2]

The same team is clear about where it does not hold. Domains that require all agents to share the same context, or involve many dependencies between agents, are not a good fit today [2]. Most coding tasks have fewer truly parallel parts than research [2].

Cognition, which builds a coding agent, reached the same conclusion from the other side. Its principle is that actions carry implicit decisions, and conflicting decisions carry bad results [5]. The simplest way to keep decisions consistent is one single-threaded agent whose context is continuous [5].

While I’m optimistic about the long-term possibilities of agents collaborating with one another, it is evident that in 2025, running multiple agents in collaboration only results in fragile systems.

Walden Yan, Co-founder, Cognition [5]

A Berkeley-led team annotated more than 1600 traces across 7 multi-agent frameworks [6] and found 14 distinct ways they fail [6]. Their summary is blunt: despite the enthusiasm, gains on popular benchmarks are often minimal [6].

A useful test before you add an agent: write down what each one owns, what it hands back and what it never sees. If you cannot, you have one agent with extra steps and a larger bill.

Control is the design decision, and the model is a detail

Notice what none of these sources argue about: which model to use. They argue about control. Anthropic reserves agents for open-ended problems where it is difficult or impossible to predict the required number of steps, and where you cannot hardcode a fixed path [1].

That is a test you can run on a whiteboard: list the steps. If you can draw them, write them in code and call the model inside the boxes. If you cannot, hand the model the wheel and pay for traces, evals and a person approving anything that sends, pays or deletes.

The teams I see struggle usually got this backwards. They started with a team of agents because it looked like good engineering, then spent a month debugging handoffs. Start with the workflow, measure it, and let the evidence promote it to an agent one decision at a time.

What to do on Monday

Write down who decides For your next AI feature, write one line per step saying whether your code, the model or a supervisor model picks what runs next.
Build the workflow first Ship the version where code holds the order, and move a decision to the model only when a real case shows the fixed path cannot handle it.
Price one request Estimate the model calls and context per run before you build, then measure it on real traffic and compare.
Rerun the same task Run each test case several times and count how often it succeeds on every run, not on one.
Keep a person on sends Anything that sends, pays or deletes waits for a human approval, whatever decides the steps before it.

Build it in the lab

For the feature you are building now, who decides what runs next: your code or the model?

Sources

Anthropic, Building effective agents (2024)
Anthropic, How we built our multi-agent research system (2025)
Sierra, τ-Bench: Benchmarking AI agents for the real-world (2024)
Hamel Husain, Your AI Product Needs Evals (2024)
Cognition, Don’t Build Multi-Agents (2025)
Cemri et al., Why Do Multi-Agent LLM Systems Fail? (2025)

Originally published at learnwithparam.com.

AI coding costs land in review, not on the invoice

Paramanantham Harrison — Tue, 13 Oct 2026 18:36:18 +0000

Your AI coding costs keep climbing, and talk of usage pricing makes next year's number harder to guess. Engineering leaders keep asking two things: will this end up costing as much as an engineer, and what are we getting for it?

Faros AI's 2025 telemetry study found that teams with high AI adoption merged 98% more pull requests, while PR review time rose 91% [1].

The AI coding bill is a small share of an engineer's pay; what decides the return is review, and without strong tests and fast feedback, more code turns into instability. To show where the money goes, we follow one change at Acme Inc: an agent writes "retry failed card payments from a queue" in an afternoon.

How we know

We read vendor telemetry studies, DORA's survey, three field experiments and practitioner surveys, kept only figures we could quote word for word from the source, and name each vendor that sells measurement or review tools.

The AI coding bill is still a small share of an engineer's pay

Per engineer, the AI coding bill is small, though the invoice is where most of the worry sits. Anthropic's figures for enterprise use of its coding agent [2] and DX's all-in estimate of seat plus token spend [3] both come to a modest monthly sum per engineer.

Levels.fyi's 2025 report puts a US software engineer's total yearly pay at about 226 thousand dollars [4]. Against that, a year at the top of the vendor's average range is a little over one percent, and a year at the top of DX's range about three percent.

A year of AI coding spend per engineer, out of each dollar a US engineer is paid [4]

Anthropic average, upper end: just over one cent
DX all-in estimate, upper end: about three cents
The engineer: the whole dollar

That is our arithmetic on total pay rather than loaded cost, so the true share is lower still. The vendor figure is an aggregate with no sample size, so treat both shares as rough; even rough, the bill is nowhere near the cost of the person.

Right now, we're not sweating the costs because we're trying to evolve best practices for the tools, but that has resulted in some devs really blowing through budget, so we may start instituting caps on spending.

Anonymous respondent, CTO, Small US company [5]

Caps may come, and a few heavy users can surprise you. At Acme, though, the token line for the payment-retry change is a rounding error on the month's invoice. On our arithmetic over a full working year, a whole active day of agent use at the vendor's average [2] costs less than one hour of the engineer's pay at Levels.fyi's figure [4].

More code arrives, but company delivery often does not move

The bill buys more code, but the evidence that it moves company delivery is thinner than the dashboards suggest. In a telemetry study of over ten thousand developers, Faros AI found that gains in team behavior do not scale when aggregated to the company [1]. Across more than 400 organizations, DX found a median PR throughput gain of 7.76% [3]: a real gain, and a modest one.

LinearB's 2026 benchmarks point to wasted output: AI-generated pull requests are accepted far less often than manual ones [6]. Code that never merges costs review time and ships nothing.

32.7% of AI-generated pull requests are accepted, against 84.4% of manual pull requests are. [6] LinearB benchmarks

It's hard to keep our CFO supportive about investing in these tools because the productivity benefits have proven difficult to conclusively prove.

Anonymous respondent, CTO, Sports-tech company [5]

Not every study agrees. DORA's 2025 survey finds a positive link between AI adoption and delivery throughput [7]; the section on tests below covers what DORA says the return depends on.

Back at Acme, the agent opens the payment-retry PR the same afternoon, and the dashboard counts a pull request, though nothing has shipped yet.

The extra output piles up in review, on your senior engineers

The extra output stalls in review. In Faros AI's data, PRs grew 154% larger on average [1], and review time rose with them.

High-adoption teams against low, Faros AI [1]

Pull requests merged: +98%
PR review time: +91%

In LinearB's separate data, AI pull requests wait 4.6x longer before anyone picks them up, though they are reviewed 2x faster once someone does [6]. Addy Osmani, who spent over 14 years leading developer experience at Google and became a director at Google Cloud AI [8], named the limit in a January 2026 essay.

When output increases faster than verification capacity, review becomes the rate limiter.

Addy Osmani, former director, Google Cloud AI [9]

PR volume rises at companies your size too: Synthesia's 118 engineers went all in on AI coding tools in November 2025 [10]. Its CTO, Peter Hill, told IEEE Spectrum that pull requests had risen 120 percent year over year [10]. He gave the volume entering review but not the hours it took.

Faros AI's head of product marketing, Naomi Lurie, argues that senior engineers absorb the review load because they catch what AI gets subtly wrong [11]. Cui and colleagues recorded seniority in the Microsoft experiment. They found the AI assistant significantly raised task completion for recent hires and juniors, but not for senior, long-tenured developers [12]. If those PRs go to seniors for review, the juniors' extra output becomes the seniors' extra review; that step is our reading, not something either study measured.

None of this is on the invoice, and no study we found prices it. Price it yourself: one hour of your staff engineer against the agent's whole token line. At Acme, the payment-retry PR is far larger than the change needed. It waits days, then lands on the one staff engineer who knows the payment flow, the same person whose juniors fill her queue.

More code merges unreviewed, and adoption tracks with less stable delivery

Under high AI adoption, more pull requests merge with nobody looking, and DORA links adoption to less stable delivery. Faros AI's 2026 report found 31% more PRs merged with no review at all under high AI adoption [11]. Faros does not say whether that is a count of merges or a share of them. On our reading, each unreviewed merge can move cost from today's reviewer to whoever handles the next incident.

31% more pull requests merged with no review at all. [11] Faros AI telemetry, high vs low AI adoption

DORA's 2025 survey finds AI adoption still has a negative relationship with delivery stability [7]. DORA reports the link, not its cause.

GitClear's analysis of changed lines shows a longer trend: refactoring shrank over the years AI tools spread [13]. The timing matches adoption, but GitClear does not show that AI caused it.

25% of changed lines were refactoring in 2021, against Under 10% were by 2024. [13] GitClear, changed lines across many repos

The evidence is mixed. Jellyfish, looking at bug tickets and reverts, found no significant relationship with AI adoption [14]. The studies measure different things in different ways, and we do not know why their results differ.

At Acme, under deadline, the second review on the payment-retry PR is waved through. A retry loop double-charges a handful of customers on a Friday. The rework and the incident are part of what the change cost.

Without tests and fast feedback, more code turns into instability

DORA's 2025 report says that without strong automated testing, mature version control and fast feedback loops, more change volume leads to instability [7]. DORA draws that from its survey, not from a controlled trial.

DORA adds that the greatest return comes from internal platforms, clear workflows and aligned teams rather than from the tools themselves [7].

Jellyfish's Nicholas Arcolano runs research at a vendor that sells engineering measurement. In a talk on his firm's data, he compared teams by how their code is spread across repos.

What’s really interesting is this highly distributed case. There’s essentially no correlation here between AI adoption and PR throughput. And actually the, the weak trend that does exist is actually slightly negative.

Nicholas Arcolano, Head of Research, Jellyfish [14]

Treat the size of any gain with care. Pooled field experiments at three companies found a 26.08% rise in completed tasks, with a standard error of 10.3%, so the true gain could be small [12].

Acme's payment-retry change, with review grown to fit

Spec: The engineer writes what the change must and must not do.
Agent writes: The agent drafts the change and its tests against the spec.
Tests gate: Nothing reaches a person until the tests pass.
Size check: Oversized changes are split before review.
Review on a target: A reviewer picks it up within the agreed pickup target.
Merge: The change merges with its review hours logged.
Survives a month: Not reverted or rewritten, so it counts as delivered.

So measure the return where it is decided. One vendor defines the unit as total fully loaded AI cost over a period, divided by the PRs that merged and survived. Survived means not reverted or substantially rewritten inside a churn window [15]. Put review hours inside that cost; our next post takes the metric apart.

Acme puts a size limit and a test gate in front of review, and sets a pickup target for agent PRs. It starts dividing all AI cost, review hours included, by changes that merged and stayed merged. The payment-retry change now reaches its staff engineer small, tested and on time.

Code was never the scarce part of your team

In a team of 20 to 150 engineers, the scarce part is the judgment of the few people who can say a change is safe. Our reading of the evidence is that every gain from cheaper code is capped by how fast that judgment can be applied. At Acme, the staff engineer who knows the payment flow was the limit before any agent arrived; the agent made the queue in front of her longer.

AI made writing cheap and left verifying slow to reach, so the cost moved to the step nobody bills for. DORA's word for it is amplifier: AI magnifies the strengths of high performers and the dysfunctions of struggling teams [16]. The evidence suggests that a team without good tests and small changes gets busier and less stable, and no faster.

So the budget question is about how you design review and testing more than about seats and tokens. The invoice is the easiest part to read and the least useful part to manage.

What to do on Monday

Write down the share Put last quarter's AI spend for each engineer beside your loaded pay. It is your baseline, and it is probably small.
Split the review clock Pull review pickup time and time in review for the last month, AI-assisted PRs apart from the rest.
Gate before a person Set a size limit and a pickup target for agent PRs, and make tests pass before anyone reviews.
Count reviews by name Count the reviews each senior engineer did last month. If one name carries the load, spread it.
Start cost per merged change All AI cost plus review hours, divided by changes that merged and were not reverted within a month.

What did your last merged change cost, counting the review?

Sources

Originally published at kaas.team.

Building a Software Factory Around Coding Agents

Paramanantham Harrison — Thu, 17 Sep 2026 16:55:31 +0000

Coding agents can write a surprising amount of code.

But getting them to reliably work on a real codebase is a different problem.

The mistake I see is treating the agent as the whole system.

A better approach is to design the system around it.

The 6 layers

I think about a Software Factory as six layers:

1. Boundary

Define where the agent can work.

This includes workspace isolation, permissions and sandbox boundaries.

For example, an ephemeral Git worktree gives an agent its own workspace instead of letting multiple tasks collide in the same working tree.

2. Context

Give the agent the context it actually needs.

Instead of continuously growing a mega-prompt, route relevant repository knowledge, engineering rules and task-specific information to the agent.

3. Skills

Separate reusable engineering workflows from individual prompts.

A skill can describe how a particular type of engineering task should be performed and reused across many tasks.

4. Execution

Give agents a controlled environment for doing the work.

This is where the execution harness becomes important. It can create the workspace, run the agent, execute commands and manage the task lifecycle.

5. Verification

Don't rely on the agent to decide whether its own output is correct.

Use automated checks such as static analysis, tests and holdout tests.

One useful pattern is:

Doer → Tester → Doer

The agent produces the change, an independent verification step checks it, and failures can feed another iteration.

6. Delivery

The final output should fit into the engineering workflow.

For me, that means getting from:

Issue → Agent → Isolated workspace → Verification → Pull Request → Human review

The goal isn't to remove engineers from the process.

It's to move engineers toward the parts where their judgment matters most.

Why this changes how we use coding agents

Once you think in layers, a lot of agent problems become easier to reason about.

Poor results aren't always a model problem.

Sometimes the agent has the wrong context.

Sometimes it has too much access.

Sometimes there is no verification layer.

Sometimes the execution environment is the problem.

And sometimes the missing piece is simply a clear delivery boundary.

That's what I've been exploring with the Software Factory approach.

I'm running a free live series on Maven covering the architecture, execution harness, automated verification and background coding agents.

The first two sessions are already available:

Design a Software Factory for Your Coding Agents
https://maven.com/p/d3588f/design-a-software-factory-for-your-coding-agents

Build Your First Software Factory Execution Harness
https://maven.com/p/aa77d5/build-your-first-software-factory-execution-harness

If you're already using coding agents on real projects, I'd love to hear how you're handling isolation, context and verification.