DEV Community: Vinny Barreca

The AI Model Is a Commodity: Why Your Moat Is the Harness

Vinny Barreca — Fri, 13 Nov 2026 23:40:59 +0000

$3 billion. That is what LangChain is worth after a $300 million Series C led by CapitalG, more than double the $1.25 billion valuation it carried a year ago on a $125 million round.

LangChain does not train a frontier model. It sells the thing you wrap around one: the gateway, the routing table, the eval set, the trace store.

The money is not in the model. It is in the layer that makes a model usable.

THE PLUMBING TRADE, BY THE NUMBERS

LangChain now claims 7,000 LangSmith customers, including Harvey, Rippling, Clay, Nvidia, Toyota and Salesforce. The pitch is one sentence: own your intelligence instead of renting it from a lab.

Cerebras reported $230.9 million in core revenue against $214.7 million expected, up 70% from $135.7 million a year earlier, and guided the next quarter to $310 million to $330 million at a 47% to 49% gross margin against a 42.1% consensus. Management said revenue triples in 2027, which implies about $2.85 billion. That is serving and silicon, not a model.

Nscale posted a $1.69 billion net loss on $307.5 million of revenue, up 1,344% year over year, with the loss widening from $1.02 billion in the prior quarter.

Perplexity shipped Decider v1.3 and Decider Flash v1.3 as open weights under Apache. On a sample of one million real requests to its Decisions API, which works out to 2,318,887 questions, the larger model agreed with frontier LLM judges 93.98% of the time. That is routing and judging, bought as a cheap model instead of a token bill.

Four companies, one direction: the capital is chasing the parts of the stack that touch every request.

WHY THE MODEL IS NOT WHERE THE MARGIN IS

A frontier model is a commodity on a 6 to 12 month clock. Whatever you pick today, a cheaper model with a matching score shows up inside a year, and the labs will cut price to keep the account.

Your harness compounds. The traces you collected last quarter are the reason you can tell a routing change from a regression this quarter, and none of that resets when a lab ships a new checkpoint.

The labs sell inference at a per token price, and price is the one lever they compete on. Everything above it, from which model gets the hard request to what counts as a good answer, is yours to own or yours to rent.

THE REQUEST PATH, UP CLOSE

Strip the marketing and the harness is three artifacts and one loop. A routing policy, an eval set, a trace store. Every request hits your gateway, the policy picks a model, the response and the model name land in the trace store, and the eval set grades a sample of what came back against criteria you wrote down.

Here is the router half of that, illustrative and shaped like a decision model call rather than any vendor SDK:

# illustrative routing policy, not a vendor SDK
policy = load("routing.yaml")

def handle(req):
    p = decider.judge(req.text, "does this need the frontier model?")
    model = "frontier" if p > policy.escalate_at else "small"
    with trace.span("route", model=model, p=p):
        return call(model, req)

That escalate_at number is your token bill in one place. Move it two points and spend moves double digits while quality moves inside the noise. This is what builders miss when they treat the model choice as the architecture. The model is a parameter. The threshold is the architecture.

FOLLOW THE QUALITY FAILURE

Google open sourced AQuA, the Ambient Quality Agent, for the part of agent work nobody puts on a slide. Agents fail silently, returning a successful-looking response while skipping a required tool call or dropping a constraint the user set three turns earlier.

AQuA runs alongside the agent in Google Cloud, outside the request path, reading telemetry from Cloud Trace, Cloud Logging or BigQuery. It samples up to 1,000 recent sessions per run through five stages: sampling, reviewing, clustering, verifying, tracking. Root cause analysis compares failing sessions against an immutable source snapshot and cites file and line ranges. It never opens a pull request on its own.

The criteria live in a plain goal.md file, and that is the decision worth copying: the standard for a good session gets versioned next to the code. The shape is roughly this, illustrative and not Google's copy:

# goal.md
 A session must
- check seat availability before quoting a fare
- carry dietary constraints into every delegation
- name the tool call behind every price it reports

 A session fails when
- the response succeeds with zero tool calls
- a constraint set on turn one is missing at turn ten

A verifier cannot reject a bad session if nobody wrote down what a good one looks like. Most teams discover they never wrote it down the day the first silent regression reaches a customer.

On 32 sessions from a multi-agent travel application that loop produced 42 findings across nine candidate clusters. The verifier threw out three unsupported clusters after checking transcripts, leaving six verified issues, among them bypassed seat availability checks and a lost vegan preference during delegation. The team's line is worth stealing: an insight is its citations, not a confidence score.

Silent quality failure is the only bug class that survives a green dashboard. Whoever owns the layer that catches it owns the renewal conversation.

THE RENT VERSUS OWN TEST

One framework, four piles, three questions, one order.

Split the bill into four piles: model API spend, serving and hardware, orchestration, and evaluation plus observability. Real accounting, not a guess from the invoice total.
If nearly all of it lands in the first pile, you are renting the commodity and ignoring the moat. That ratio is the whole diagnosis.
Ask each vendor one question: what happens to our agents if your model API is deprecated next quarter? The answer tells you whether you bought a harness or a habit.

The ordering rule is the valuable part. Evaluation goes first because it is the only pile that tells you whether the other three work. Routing comes second, since a good eval tells you which requests deserve the expensive model. Serving comes last, once you know the traffic you pay to serve.

THE BEAR CASE, STEELMANNED

Plumbing is capital hungry, and Nscale is the proof. A $1.69 billion quarterly loss on $307.5 million of revenue is a company buying demand with borrowed money. Microsoft walked away from a planned data center in England over electricity access and declined to renew a Texas contract. Anthropic exercised an option in late September for more capacity.

So the trade has two halves. The margin is in the plumbing, and so is the balance sheet risk. If capital tightens, the layer consolidates around whoever holds contracted demand, not whoever holds the best deck. Owning your intelligence does not mean owning a datacenter.

WHAT TO DO TODAY

Map this quarter's AI spend across the four piles. Write the percentages down.
Ask your model vendor the deprecation question in writing.
Write the goal.md for your worst agent before you shop for a better model. Ten criteria will teach you more than ten benchmarks.
Stand up one eval or routing pilot before the renewal date on the calendar.
Put the Q1 2027 Cerebras and Microsoft revenue line on a watchlist. It is the cleanest canary for whether this trade still has buyers.

THE UNCOMFORTABLE QUESTION

If the model is the commodity, what exactly are you paying your AI budget for, and who is capturing the margin you assume is yours?

You already know the model is a parameter. Here is what that costs in practice: Same task, same model, same tools: 36 requests and 3.1M tokens direct, 2 requests and 61K. The threshold and the routing table, not the checkpoint, are where the bill actually moves.

Stop Decorating Invoices: Why Multi-Agent Systems Fail the Token Multiplier Test

Vinny Barreca — Thu, 12 Nov 2026 00:52:26 +0000

4.7 times the tokens. That is what every coordinated arm paid in a controlled study of thirteen enterprise workflows, and the success gains were usually small and rarely significant.

The setup is what makes the number worth your attention. Same model. Same tasks. Same tools. Only the architecture changed.

Then came the twist. A redesigned multi-agent build hit 75.6% success against a 67.4% baseline, the only arm in the study to significantly beat the single agent. The architecture did not win that round. The implementation of it did.

If your team is about to add a second agent, read the ledger below first.

WHAT THE STUDY ACTUALLY DID

The paper is Agent Architecture Cost-Success Trade-offs: A Controlled Evaluation across Enterprise-Style SOPs, on arXiv as 2611.12953, submitted 7 November 2026 by Subhrangshu Nandi and thirteen co-authors. Read it before you argue with anything below, because the details are the argument.

Most agent benchmarks vary everything at once. They swap the model, swap the harness, swap the task set, and then publish a score. That measures confusion, not architecture.

This study held the model, the tasks, and the tools fixed and varied one thing: the shape of the workflow. It compared a function-calling single agent, a skill-augmented agent, centralized multi-agent orchestration, and a triage swarm, eight arms in total, across 13 enterprise SOPs.

An SOP here is a standard operating procedure, the long branching workflow a company actually runs: intake, checks, tool calls, handoffs, sign off. The kind of process with twelve steps and three places it can dead-end.

The authors also built a skill definition quality rubric and used it to assemble two matched corpora, one with high-quality skill definitions and one with deliberately degraded ones.

THE LEDGER: WHAT COORDINATION COSTS

Every coordinated arm, multi-agent or skill-augmented, spent 1.4 to 4.7 times the tokens and 2.6 to 5.8 times the latency of the single agent.

For gains that were usually small and rarely significant.

On any given SOP, success ranged from under 10% on the hardest to near 100% on the easiest. In aggregate, no architecture was statistically separable. Added structure only plausibly paid off on complex branching SOPs, and the authors call that a directional trend rather than a significant one.

This is the part that hits your bill and not your benchmark. Token accounting per request hides the whole problem now. Coordination means an orchestrator that talks to two workers that call tools that return to the orchestrator. Every hop re-sends context. Your dashboard shows a fleet of cheap calls. The invoice shows the multiplier.

THE ARCHITECTURE MULTIPLIER LEDGER

Name the thing so your design review can point at it. The Architecture Multiplier Ledger is four steps.

Instrument tokens and latency per SOP on your current single-agent before anything new lands.
Classify each SOP by branching depth and length, because those are the only shapes where a gain is even plausible.
Add coordination only where a measured gain clears the recorded multiplier.
Re-run the ledger after every skill corpus change, since definition quality moves success more than architecture does.

The ordering rule is the whole framework: record the multiplier before you add the structure. If the gain does not clear the multiplier, the structure is a donation to your inference bill.

An illustrative version of the measurement, not a result from the paper:

def ledger(sop, baseline, candidate):
    b = baseline(sop)                      # one run, single-agent
    c = candidate(sop)                     # same SOP, coordinated build
    token_mult = c.tokens / b.tokens
    latency_mult = c.latency / b.latency
    return {
        "sop": sop,
        "token_mult": round(token_mult, 2),
        "latency_mult": round(latency_mult, 2),
        "success_gain": c.success - b.success,
        "clears": c.success - b.success > token_mult,
    }

Run it per SOP, not per system. One average across thirteen workflows is how a bad arm hides.

WHY DEFINITION QUALITY BEAT ARCHITECTURE

High-quality skill definitions lifted success from 67.4% to 73.1%. Degraded definitions fell below the baseline.

Same model, same tools, same SOPs. The only variable was how well the skill was written. That is a bigger move than most teams get from their entire orchestration layer, and it costs a documentation pass instead of a second agent.

This is the same law measured a second time, and it lines up with the earlier note on tool descriptions costing 66% accuracy. The words you put in front of the model are the product. The topology around them is overhead.

The practical consequence is uncomfortable for people who think of prompt files as scratch work. A skill corpus is a testable asset. It belongs in version control, with a rubric, and it should be scored the same way you score code before a release.

THE REDESIGN RESULT

The sharpest finding is not that multi-agent lost. It is that a plausible multi-agent implementation lost to the single agent, and a targeted redesign of that same implementation became the only arm to significantly beat the baseline at 75.6%.

One build failed. The same build, fixed, won. Nothing about the category changed between those two rows.

So the variance inside one architecture is larger than the gap between architectures. Your team is arguing about a category while the outcome sits in the build. That is why the debate never resolves.

And the tooling is following the bill rather than the benefit. A Claude Code hooks module called Reinwatch, posted the same morning it was written, reports per-agent token counts, and can stop a single runaway agent inside a running workflow. People ship products for problems that already hurt.

WHAT TO DO TODAY

[ ] Turn on per-agent token and latency accounting for every agent you run, in your own harness if you have nothing off the shelf.
[ ] Baseline your single agent on your five most used SOPs before you add a second one.
[ ] Audit your skill definitions for staleness and vagueness, because that is where the points are.
[ ] Kill one coordinated arm you cannot justify with a measured gain. You already know which one.
[ ] Put the multiplier ledger in your design document template this week.

THE UNCOMFORTABLE QUESTION

If your team's architecture debates never end in a measured token multiplier, are you engineering a system or decorating an invoice?

Same setup, different layer: in Programmatic Tool Calling: Why Your AI Harness Beats Your Model Choice, the same task, same model, and same tools run 36 requests and 3.1M tokens through one harness and 2 requests and 61K through the other.

Your LLM Capacity Plan Is Lying to You: Inside vLLM Overload Traces

Vinny Barreca — Tue, 10 Nov 2026 02:00:23 +0000

During an overload experiment on vLLM, the engine's own prompt token counter missed a third to half of every prompt token it computed. Requests that were later aborted had already burned 79 to 85 percent of their prefill work before anyone cancelled them.

That is not a bug report. That is your capacity plan sitting on a meter that undercounts by half during exactly the incident you plan for.

We covered the client side of retries in an earlier note, where idempotency and reconciliation decide whether a retry is a second attempt or a second charge. This is the other end of the wire: what the engine does when every client retries at once.

THE EXPERIMENT THAT DID NOT GO AS PLANNED

Two researchers ran overload experiments on vLLM with three open models and recorded scheduler traces of one of them. A scheduler trace is the record that ties each client attempt to the execution it created inside the engine. That instrument matters more than the headline. Without it you see requests arrive and finish. With it you see the work a dead client left behind.

The assumption under test was the fix most teams ship first because it looks free: propagate the cancellation and the wasted compute goes away.

It did not. With retries on, cancellation alone failed to restore service in any run. Only 12 to 38 percent of the calls that arrived after the overload were delivered.

Cancellation is free, and you get what you pay for.

WHY CANCELLATION DOES NOT UNBURN TOKENS

The traces show why. Retries that vLLM ran typically started prefill less than half a second before they timed out. Prefill is the prompt processing that happens before the first token comes out, and for a long agent context it is most of the bill. It was nearly finished when the deadline fired.

So the client gave up, the engine kept computing, and the cancel signal arrived after the money was spent.

Then the accounting failure. The engine's prompt token counter missed a third to half of all the prompt tokens it computed during the incident. Your autoscaler, your budget alerts, your per tenant cost report and your capacity plan all read from that counter. They all read low at the worst possible moment, and they all read low in the same direction.

THE OVERLOAD TRIAGE RULE

Three remedies get sold for retry storms, and they fail in different ways. Pick by which failure you can afford.

Cancellation stops new waste. It does not recover work already spent.
A retry budget stops amplification. It does so by refusing service to some callers.
Request coalescing stops duplicate work. It delivered every call from the overload onward, some of them late, unless cancellation had already aborted the running execution.

The order matters, and that is the part most teams get backwards. Coalescing wins only while the original execution is still alive, so cancellation running ahead of it is what turns a late answer into no answer. Coalesce first, budget second, cancel last.

That order is the whole interceptor. The shape below is illustrative, not the paper's code, but every line maps to one of the three remedies:

# gateway interceptor, illustrative shape only inflight = {}

def intercept(req):
    key = (req.prompt_hash, req.model, req.deadline_bucket)
    if key in inflight:
        return inflight[key].subscribe()   # coalesce: one execution, many waiters
    if not retry_budget.take(req.client):
        return shed(req)                   # budget: refuse on purpose, not by luck
    ex = engine.submit(req)                # prefill starts burning tokens here
    inflight[key] = ex
    return ex.result(timeout=req.timeout)  # no cancel call, and that is deliberate

Leaving out the cancel call is the design decision, not an oversight. Kill an execution that other callers are attached to, and you have converted a late answer into no answer for all of them.

Pick your remedy by which failure you can afford: waste, refusal, or latency.

WHY AGENTS MAKE THIS THE COMMON CASE

DeepSeek disclosed that its Harness agent app was consuming roughly 92 trillion tokens per day at 1.54 million daily users, which works out to about 60 million tokens per user per day. Chat traffic never produced a number like that. An agent with a retry loop multiplies request volume and prompt length at the same time, because every retry carries the same long context into prefill.

Load shapes are also diverging. None of the top ten open source models on OpenRouter share an architecture, and average prompt lengths differ by region, 97k tokens against 70k. When the shape of the load keeps moving, the overload point moves with it, and the timeout you tuned last quarter is now a retry generator.

The overload path is not the edge case you deprovision capacity for. When your users are agents holding timeouts, it is the product path.

WHAT TO DO TODAY

Stop trusting the engine's prompt token counter during an incident. Log scheduler level executions, not just completed requests.
Measure prefill completed before timeout. If retries start prefill under a second from their deadline, your timeout is your retry policy.
Build coalescing at the gateway, keyed on the request, so duplicate calls share one execution instead of buying one each.
Cap client retries with a budget and make the budget visible to the server, so shedding is a decision rather than a coincidence.
Run an overload drill on purpose and record delivered call percentage, not average latency. Average latency hides undelivered calls completely.

That last one is the measurement that would have caught this. Every team watching this incident had a latency graph. The graph that told the truth was the one counting calls that never got an answer.

THE UNCOMFORTABLE QUESTION

Your dashboards showed green through the last incident because the counter feeding them undercounts by half under load. How much of your capacity plan is built on a meter you have never verified during an overload?

Anthropic just paid Akamai $11.6 billion over seven years, and the bet is on CPUs, not GPUs

Vinny Barreca — Sat, 26 Sep 2026 19:10:59 +0000

Everyone argues about which GPU buys the frontier. Meanwhile, the biggest cloud deal of the year is for the boring chip that runs your agent's tool calls, browsing, and code execution.

Agents do not live inside the model forward pass. They live in the harness around it, and that harness is CPU work. That is why a lab would lock up seven years of general-purpose compute while everyone else fights over HBM.

We covered this before it was news: the agent workload already moved to the CPU.

I Scanned 300 Small Business Domains. 243 Had Broken Mail Records.

Vinny Barreca — Sun, 20 Sep 2026 16:37:35 +0000

I ran a mail-record scanner over 300 small business domains, the kind of local shops that send invoices, quotes, and appointment reminders out of their own domain.

243 of them, 81 percent, had at least one defect.

Not a judgment call. A measured defect: no SPF record, an SPF ending in ~all, a DMARC record that enforces nothing, or a DKIM key that has been revoked. The breakdown: 120 domains with no DMARC record at all, 112 with SPF ending in ~all, 73 running DMARC at p=none, 66 with no SPF record at all, and 48 with a DMARC record that collects no reports.

Eighty-one percent. That is not a rounding error. That is the default state of small business email.

THE ONE RECORD NOBODY PUBLISHES

Here is the one I see most and the one almost nobody understands.

A restaurant domain had this in DNS:

v=SPF1 include:spf.easywp.com ~all

and this at _dmarc:

v=DMARC1; p=none;

The SPF is fine. The DKIM is fine. But the DMARC policy says p=none, which tells every receiver on earth: if mail claiming to be from this domain fails both checks, do nothing. Deliver it anyway.

p=none is not enforcement. It is a shrug written in DNS.

The owner thinks they have DMARC because the record exists. They have the appearance of DMARC. Any spammer with a cheap mail server can send "invoices" from their domain, and every receiver will wave it through, because the record they published literally instructs receivers to accept the failure.

Worse: this particular record had no rua tag, so even the reporting, the only reason p=none exists as a testing stage, was switched off. The domain was reporting nothing to nobody.

THE RECORD THAT DOES THE OPPOSITE OF WHAT YOU THINK

Second most common: SPF ending in ~all.

v=spf1 include:secureserver.net ~all

~all is soft fail. It tells receivers: mail from an IP not on this list is suspicious, but you should probably accept it anyway. That is a suggestion. Receivers treat a soft fail as a yellow light and wave most of it through while flagging you for suspicious behavior.

The fix is -all. Hard fail. Any mail from an unauthorized IP is refused.

So why does ~all exist everywhere? Because owners are afraid of locking themselves out. Legitimate fear, wrong remedy. You do not flip to -all on day one. You publish DMARC with a rua reporting address first, at p=none, and let the reports tell you every service actually sending as your domain. You will find one or two you forgot about. Then you authorize them, and then you flip.

The staging is the fix. Nobody does the staging, so everybody sits at ~all forever.

THE DOMAIN WITH NOTHING

And then there is the domain with no SPF and no DMARC at all.

Empty. Both records. No MX record either, so the domain receives no mail and sends with no authentication whatsoever.

Anyone on earth can send mail as this domain. Your bank, your customers, your own employees, and nobody receiving it has any instruction on what is real. That is not a deliverability problem. That is an open door.

This was not a rare find. 66 of the 300 domains had no SPF. 120 had no DMARC. If you are one of them, the question is not whether someone is spoofing you yet. The question is what stops them when they start.

WHY MAIL-TESTER SAYS 10 AND GMAIL SAYS SPAM

The classic trap. You run mail-tester, it says 10/10, and your invoices still land in junk.

Mail-tester grades the message. Gmail grades the domain. A perfect score on a single test message tells you nothing about your domain reputation, and it says nothing about whether your DKIM selector is actually the one your provider signs with.

Half the "10/10 and still spamming" questions I see are a provider mismatch. The domain carries DKIM records for a platform the owner stopped using two years ago, and there are no records for the platform they actually send from today. The stale sender sits in SPF, burning trust. Nobody notices because nothing enforces alignment. A p=none DMARC hides it by design.

You cannot diagnose that from a message grader. You diagnose it from the records.

WHAT TO DO TODAY

Three checks, ten minutes, no tools you have to pay for.

One. Look up your domain's TXT record. If SPF does not exist or ends in ~all or +all, that is your problem. +all is worse than missing: it authorizes the entire internet.

Two. Look up _dmarc.yourdomain.com. If the record does not exist, publish one at p=none with a rua pointing at a mailbox you actually check. That is behavior-identical to what you have now; it cannot break sending, and it starts the reporting engine.

Three. Once a few weeks of reports show every legitimate sender aligning, tighten. ~all to -all. Then p=none to p=quarantine. Then, after another clean window, p=reject. Each stage is its own decision.

Never jump straight to p=reject. That is how a fix becomes an outage, and it is the reason most owners freeze at p=none and call it done.

THE UNCOMFORTABLE QUESTION

I fixed my own domain the same way: staged DMARC, watched the aggregate reports from Google land in my inbox for a day, confirmed every message passing dkim and spf with zero failure reasons, then tightened to -all and p=reject. The Google report confirmed the fix end-to-end. That is the standard, and it is not a high bar.

So: is your domain one of the 243 or one of the 57?

You do not know until you look. Go look.

If you would rather hand the DNS surgery to someone who does it every day, I fix these records the same day, flat $500, and verify the result with real delivered mail: PhantomByte Email Deliverability Rescue Service

Your Agent's Tool Descriptions Are Costing You 66% Accuracy

Vinny Barreca — Tue, 15 Sep 2026 18:56:49 +0000

A $4 experiment rewrote three strings per MCP tool and moved SQLite success from 34% to 100%. The model never changed.

The failure was never the model.

SQLite strict success rate: 34%. That is a failing grade on a benchmark nobody was running.

Toolmetry, an open-source project that measures how well agents use MCP tools, took existing MCP server tool descriptions and had an LLM rewrite them. The strings that tell an agent what each tool does and how to call it. That was the entire change.

The model did not change. The harness did not change. Only the descriptions changed.

After the rewrite, SQLite hit 100%. Total API spend for the whole experiment was $4.

This is an interface problem, not a model problem. Tool descriptions are the contract between your agent and the outside world. When the contract is ambiguous, the agent breaks the terms every single time.

The three failure archetypes

Toolmetry found three patterns that were killing agent success rates before the rewrite.

Wrong tool confusion. Two tools in the same MCP server have overlapping descriptions, so the agent picks Tool A when it needs Tool. B. It looks like a bug. It is bad documentation.
Ritual extra calls. The agent makes a call it believes is a prerequisite, burning tokens and latency for nothing.
Deprecated parameter patterns. The description references parameters or usage patterns that no longer work. The agent follows the instructions faithfully and produces errors.

Each one is fixable with better words. Not better models. Better words.

Wrong tool confusion in practice

Before the rewrite, the SQLite server's read and write query tools were described almost identically. Most failures were the agent calling a read tool when it needed a write tool.

Two tools that sound similar to a human sound identical to an agent. After the rewrite, the query tool description stated explicitly what it could and could not do. Success jumped to 100%.

If two tools could be confused by a smart human, they will be confused by your agent.

Ritual extra calls and the implied prerequisite

Agents were calling a list_tables function before every query because the query tool's description said "first enumerate available tables." Implied, not required.

Before:

Tool: query

Description: Execute an SQL query against the database. First enumerate
available tables using list_tables, then construct your query against
those tables.

The agent called list_tables before every query, even when it already knew the schema from the previous call. Two extra API calls per interaction, doubled latency, zero benefit.

After:

Tool: query

Description: Execute an SQL query against the database. Available tables:
users, orders, products, invoices, and audit_log. Use standard SQL syntax.
This tool handles both reads and writes. No need to call list_tables first.

The ritual calls stopped immediately. The table list was right there in the description. No extra round trips, no wasted tokens.

Remove implied prerequisites from your descriptions. If a step is not required, do not imply that it is.

Deprecated parameters

The git server had deprecated parameter references baked into its descriptions. Agents were passing old parameter names and getting failures. The documentation was lying about how the API worked.

After the rewrite, all parameters matched current names. Success went from 75% to 96.7%.

Version your tool descriptions. When you change an API, update the description in the same commit. A stale description is worse than no description.

With no description, the agent guesses. With a stale description, the agent confidently does the wrong thing.

The before and after numbers

Server Before After Improvement

SQLite 34.0% 100% +66.0 points
Memory 61.8% 96.4% +34.5 points
Git 75.0% 96.7% +21.7 points

$4 of API spend for the entire run.

Now compare that against upgrading to a frontier model. A cheaper model with broken descriptions still fails. An expensive model with broken descriptions still fails. The model is not the bottleneck. The interface is.

Audit your MCP server in under an hour.

This costs less than $5 in API spend and takes about an hour.

List every tool in your MCP server and print each description.

Read pairs of descriptions side by side. If two could be confused by a smart human, they will be confused by the agent.

Check for implied prerequisites. Does any description say "first do X" when X is not actually required?

Check for deprecated parameters. Diff your current API against the description text.

Rewrite the descriptions with an LLM. Feed it the tool name, the current description, and the real API signature. Ask for a clear, non-overlapping description.

Re-run your agent test suite and compare success rates.

Toolmetry ships three commands to automate the loop: measure, optimize, and proxy. The proxy rewrites descriptions on the fly, so you can test improvements before committing anything.

Why this matters more than model selection

This is the fourth data point in a pattern that keeps repeating. Three separate papers already showed orchestration design beating model selection by 10x on token cost.

Structural scaffolding reduces failure rates under a fixed model. Graph-structured context beats flat text. And now tool descriptions alone move a server from 34% to 100%.

How you structure the agent's control flow, context, and tool interfaces matters more than which model you use. That is a direct challenge to the scaling; all you need is orthodoxy.

The competitive advantage is shifting from model ownership to orchestration design. The people who understand that will win. The people who keep throwing bigger models at broken interfaces will keep getting 34% success rates and wondering why.

The bottom line:

Your tool descriptions are three strings per tool. They cost nothing to write and everything to get wrong.

Wrong tool confusion, extra ritual calls, and deprecated parameter patterns exist in every MCP server that has never been audited. They exist in yours.

Audit them today. It takes an hour. It costs $4. It might double your agent's success rate.

Originally published at articles.phantom-byte.com.

PhantomByte teaches you to build real AI infrastructure: local AI stacks, autonomous agents, multi-agent orchestration, and custom tools. Step-by-step tutorials you download, follow, and deploy.

The $750 Copilot: Why Your AI Dependency Just Got a Price Tag

Vinny Barreca — Mon, 01 Jun 2026 07:44:18 +0000

Your GitHub Copilot bill just went from $29 to $750/month.

Microsoft taught developers to "vibe code" (write less and let AI do the work).

Build entire workflows around that subsidy.

Then they installed a toll booth.

This is not a pricing change. It is a dependency trap snapping shut.

The $750 Copilot: the first honest price tag in a subsidized industry. Full breakdown and the local-first alternative