AI sales agent test: four subagents vs one loop with skills

This post walks through a small test of an AI sales agent built two ways, a coordinator with three specialist subagents and a single loop that loads skills, so that by the end you can rebuild the same comparison on your own agent: write the desk as data, share the tools between both designs, grade every answer against a fixed key, run it offline first, and read the results without fooling yourself.

The trigger was Anthropic's Applied AI team publishing Building agents: When to fold subagents into your main agent loop on 22 October 2026. Their claim, from their own comparison across several enterprise deployments, is that a single agent with skills frequently beats both one-prompt-for-everything and a subagent design on quality, often at lower cost and latency. They didn't publish the numbers. Their method advice is concrete though: build a small end-to-end eval first (ten golden examples is enough), fold one subagent at a time, run each task more than once, and look at which tasks flip.

So I had Claude Code build exactly that on a quote desk, and a GitHub Actions workflow made the API calls.

The desk is one JSON file

Everything the agents can know lives in one file: products, customers, stock, rules and the test emails. The company is Nordlys Kontor ApS, an invented Danish office-furniture supplier, and its customers and emails are invented too.

Four products, priced in Danish kroner: an Ergo 2 office chair (DKK 2,400 list), a sit-stand desk (4,800), a monitor arm (600) and an acoustic panel (1,000). Six CRM customers. Stock in Vejle and Malmö. Trimmed from the real file:

{
  "catalog": {
    "ERGO-2": {"name_en": "Ergo 2 office chair", "list_dkk": 2400},
    "ARM-1": {"name_en": "Single monitor arm", "list_dkk": 600}
  },
  "stock": {"ERGO-2": {"...": "units in Vejle and Malmö, next batch date"}},
  "customers": [{"id": "K-1187", "language": "en", "framework_agreement": false,
                 "payment_terms": "...", "oldest_overdue_invoice_days": 0}],
  "procedures": {"pricing": "...", "delivery": "...", "terms": "..."},
  "tasks": [{"id": "T02", "lang": "en", "customer": "K-1187", "lines": {"ERGO-2": 120},
             "trap": "quantity tier at 100", "body": "..."}]
}

The rules are written as procedure text rather than code, because both designs have to read them as instructions. Per line: 0% for 1 to 24 units, 5% for 25 to 99, 10% for 100 or more; a framework agreement adds three points. Freight is DKK 495 under DKK 10,000 of goods. Vejle delivers in 3 working days, Malmö in 7, the next batch adds 3, and the whole order ships at once. An invoice more than 30 days overdue means prepayment, a new customer gets net 14, otherwise net 30. Escalate to the sales manager on an extra discount ask, a product not in the catalogue, or a missed deadline.

Each of the twelve emails (six Danish, six English) has one trap: the 100-unit tier, a quantity written as a word ("fyrre", forty), stock split between Vejle and Malmö, stock short in both, a framework agreement, a 15% ask, a new customer, two lines plus an agreement plus a 40-day overdue invoice, an invoice only 12 days overdue, a deadline missed by one day, an agreement plus the 100 tier plus a 20% ask, and a Danish customer writing in English with an order just over the freight line.

If you build your own, spend your time here. Traps that sit on a rule boundary are where agents disagree with each other.

Shared tools in desk.py

desk.py holds the data tools both designs use: find_customer, get_price and check_stock, plus submit_quote at the end. They read the JSON file, nothing else. If the tools hit a live CRM, the answer key drifts and you can't rerun the test next month and compare.

Each tool is the usual Anthropic tool schema next to the function it calls. From desk.py, trimmed:

"get_price": {
    "description": "List price of one product by SKU, in DKK excluding VAT. Catalogue: ...",
    "input_schema": {"type": "object", "properties": {"sku": {"type": "string"}}, ...},
    "fn": lambda a: get_price(a.get("sku")),
},

Both designs get the same tool functions. The only thing that changes between them is who holds the tools and where the procedure text sits.

Two designs in api_job.py

Both run on Claude Sonnet 5.5 at medium effort.

Design A, four agents. The coordinator gets three tools: ask_pricing, ask_delivery and ask_terms. Each takes a question string, and calling one starts a separate specialist loop with its own context, its procedure as the system prompt and its own data tools. The specialist never sees the customer's email, only the coordinator's question, and it answers in free text. The coordinator has no catalogue of its own. Specialists were told to answer briefly with the numbers.

Design B, one agent. One loop with a load_skill tool that returns the same procedure text the specialists got as system prompts, plus the same data tools and submit_quote:

LOAD_SKILL = {
    "name": "load_skill",
    "description": "Load one of your skills and read its full procedure. Skills: ...",
    "input_schema": {"type": "object",
                     "properties": {"name": {"type": "string", "enum": list(D["procedures"])}},
                     "required": ["name"], "additionalProperties": False},
}

Keeping the procedure text identical across the two designs is what makes the comparison fair. If you rewrite the pricing rules while folding them into a skill, you're testing two changes at once.

The grader checks eight things against a fixed key

Every quote is graded by a script, not by a model, on eight checks:

1. the total
2. the order lines, including product code and freight
3. the delivery date
4. the payment terms
5. escalation to the sales manager, or not
6. the reply language
7. the total stated in the reply text
8. the date stated in the reply text

The key is not stored in the emails. expected() computes it from the written procedures, and grade() compares, trimmed:

def grade(task, sub):
    e = expected(task)   # the answer key, computed from the written procedures
    return {
        "total": sub.get("total_dkk_ex_vat") == e["total_dkk_ex_vat"],
        "lines": got_lines == want_lines and sub.get("freight_dkk") == e["freight_dkk"],
        "delivery_date": sub.get("delivery_date") == e["delivery_date"],
        # ... five more, each a plain comparison
    }

A fixed answer key is the whole point. With an LLM judge, a wrong total phrased confidently can pass, and the judge itself varies between runs. With a key, a quote is right or it isn't, and you can look at exactly which check flipped. The kit also ships an independent verifier alongside the grader.

Run it offline first with FOLD_FAKE=1

Set FOLD_FAKE=1 and the job runs against a fake model instead of the API. It costs nothing and proves the plumbing: the desk loads, the tools return what you expect, quotes get submitted and graded, results get written. Only when that's clean do you add your own API key and run it for real. In the kit that is python3 desk.py to selftest the key and the grader, FOLD_FAKE=1 LAB_OUT=out python3 api_job.py for the offline run, and python3 score.py out/runs.jsonl to score it. verify.py recomputes every number without using desk.py or score.py.

Each email ran three times per design: 72 quotes, 364 model calls, zero errors, USD 2.19 at list price (USD 1.48 for the four agents, USD 0.71 for one).

The numbers

All eight checks: one agent 36 of 36, four agents 29 of 36. Counting only what a customer reads (everything but the product code): 36 against 35.

Model calls per quote: the single agent made exactly 2 every time. Call 1 loaded all three skills and ran every lookup in one parallel batch; call 2 submitted. The four agents made 8 (32 quotes) or 9 (4 quotes): one or two coordinator calls to send the questions, two per specialist (once three), and one coordinator call to write the quote.

Per quote, medians for seconds and tokens, the average for cost:

	Four agents	One agent
Seconds	13.9	7.3
Tokens	12,704	6,239
Cost	4.1 US cents	2.0 US cents

That's about USD 41 against USD 20 per 1,000 quotes. The seconds are approximate: they were measured on GitHub's servers, six quotes at a time, interleaved between designs, and the job capped concurrent calls at 10 locally, a cap the parallel specialists hit more often.

One result runs against the usual pitch for skills. The single agent loaded all three skills on every quote, so progressive loading saved nothing here. The gain came from one context and one batch of lookups.

What failed, and where

A multiplication slip in a handoff. On T02, 120 chairs, the coordinator asked the pricing specialist: "What is the unit price, line total, and any freight?" The specialist got the unit price right, DKK 2,160 after 10%, then wrote Line total: 2,160 × 120 = 288,000 DKK. The right answer is 259,200. It made that slip in runs 1 and 3. In run 1 the coordinator redid the sum and sent 259,200. In run 3 it copied 288,000 onto the quote, next to the 2,160 unit price, DKK 28,800 too high. The single agent did the sum itself in all three runs.

Models slip on arithmetic now and then. The design problem is that checking was optional, and the coordinator took the option differently in two runs of the same input.

Missing product codes. The coordinator has no catalogue, so the code can only arrive in the specialist's prose, and it arrived exactly when the coordinator's pricing question asked for it. 29 questions asked for the SKU and all 29 quotes kept it. 7 didn't ask, and the order line said "Ergo 2" instead of ERGO-2, or "Single monitor arm" instead of ARM-1. All seven were in T02, T08 and T11, the emails with the biggest discounts. The single agent: 0 of 36.

What I would change in the four-agent design

Anthropic's advice for subagents is a shared response schema and having the main agent verify a subagent's conclusion against real data. Applied to this desk, my reading of that is two changes.

A schema for every specialist. Make the specialists return structured output instead of prose, with fields like code, quantity, unit_price_dkk, line_total_dkk and freight_dkk. That would very likely fix all seven missing codes, because a required field can't be forgotten.

The coordinator recomputes totals. A schema won't catch 2,160 × 120 = 288,000; it'll carry the wrong number in a perfectly valid field. Have the coordinator, or better a few lines of plain code after it, recompute quantity * unit_price_dkk and refuse to submit on a mismatch. That makes run 1's behaviour a rule rather than luck.

Or fold it, which is what Design B is. On this desk none of Anthropic's three reasons to keep a subagent applied: context bloat, parallel work in throwaway contexts, or trouble picking tools.

What the test doesn't tell you

One desk, one model, three runs per email, invented data, a script grader. Specialists never saw the email; a design that forwards the email to every specialist wasn't tested. The specialists' instructions never mentioned codes; only the coordinator's question did, when it remembered. Treat it as a method you can copy and a pair of failure modes to look for, not as proof that one loop always wins.

Running the test on your own AI sales agent

The free kit with the desk JSON, both designs, the fake model, the grader, the verifier and all 72 recorded quotes runs with your own API key and doesn't talk to a CRM. Swap the products, customers, rules and emails for yours, keep the procedure text identical between designs, run with FOLD_FAKE=1 first, then for real, three times per email. For migrating a production agent, Anthropic's single-agent-loop-kit is the reference implementation.

The full write-up with an interactive demo of every email on both desks is the English article on four AI agents against one for quotes, and there's a Danish version of the article.

I'm Kim Olsen, and I run a one-person Claude specialist studio in Copenhagen. If you'd rather have this built for your sales team, with rules as skills, systems as tools and an answer key that tells you whether a change helped, see how I build internal AI tools.

Comments

[Kent Bodrov]

Since you already call out the email-visibility limit, I'd make that the next controlled comparison: cross shared email/no shared email with prose/typed handoffs, keeping four agents and the same grader. Then compare that best four-agent result with the single loop. It would distinguish an architecture cost from a lossy handoff. I'd retain the SKU check in deployment acceptance, even if it hardly changes the customer-readable score: fulfillment still needs that identifier.