My tool-calling agent looked amazing in demos and then double-charged people

I knew I was in trouble when the demo agent did exactly what I asked on Friday, then did it twice on Monday.

Same prompt.
Same API.
Same nice-looking trace.

But this time the second tool call hit a real downstream action and charged someone twice.

That was the moment I stopped treating agents like smart prompts and started treating them like flaky software integrations with an LLM attached.

That shift changed how I think about AI agent QA.

Most teams still test agents like it’s 2023. They run a few happy-path prompts, watch GPT-4o or Claude call a function correctly, maybe post a screen recording in Slack, and call it done.

Then production shows up with malformed JSON, retries, stale memory, a Stripe 500, a HubSpot timeout, and duplicate actions nobody designed around.

At that point the model didn’t "go rogue."
Your QA failed.

The demo was never the hard part

Getting a tool-calling agent to look smart in a demo is easy now.

• n8n makes it easy
• OpenAI makes it easy
• Anthropic makes it easy

You wire up an HTTP Request Tool, maybe a Custom Code Tool, maybe let the agent call a full n8n workflow through the Call n8n Workflow Tool, and an hour later you have something that can search, summarize, enrich, and post to Discord like it has a tiny soul.

But demos hide the real problem: the outside world behaves.

Production is where:

• auth expires
• APIs return partial payloads
• retries happen
• memory carries stale context forward
• downstream systems mutate state twice

A lot of so-called reasoning bugs are just state bugs wearing a trench coat.

The ugliest failures I saw were not mystical LLM failures. They were boring:

• bad parameters
• duplicate requests
• missing fields
• half-finished tool round trips

That’s actually good news, because boring failures are testable.

OpenAI Structured Outputs changed the baseline

This is the part I think a lot of teams still haven’t updated on.

OpenAI Structured Outputs changed what counts as acceptable tool-call behavior.

If you set strict: true in the function definition, OpenAI reported that gpt-4o-2024-08-06 hit 100% schema adherence on complex JSON-schema evals, versus less than 40% for gpt-4-0613.

That’s not a small quality bump.
That’s a QA philosophy change.

If your tool-calling agent still emits malformed arguments under a strict schema, I would stop calling that “LLMs being weird” and start calling it a regression.

Here’s the shape of a strict function definition:

{
  "type": "function",
  "function": {
    "name": "query",
    "description": "Execute a query.",
    "strict": true,
    "parameters": {
      "type": "object",
      "properties": {
        "table_name": {"type": "string"},
        "columns": {
          "type": "array",
          "items": {"type": "string"}
        }
      },
      "required": ["table_name", "columns"]
    }
  }
}

That should immediately suggest a regression test:

• same prompt
• same expected action
• once with loose JSON handling
• once with strict schema enforcement

If strict passes today and fails next week after a prompt tweak or model swap, you caught a real bug before a customer did.

Strict schemas do not solve:

• wrong tool choice
• stale memory
• bad business logic
• unsafe retries

A tool call can be perfectly valid JSON and still be a terrible decision.

But once argument shape becomes reliable, the remaining failures get easier to isolate.

Why my agent did the same thing twice

Because I let it.

Anthropic’s tool-use docs are very clear about something demo builders ignore: Claude can emit one or more tool_use blocks in a turn.

If your app treats that as a cute implementation detail instead of an orchestration contract, duplicate side effects are your fault.

And if your downstream API is not idempotent, repeated calls are not a model alignment problem. They are an integration design problem.

Anthropic even gives you a lever for this:

{
  "tool_choice": {
    "type": "auto",
    "disable_parallel_tool_use": true
  }
}

If I’m touching anything that:

• charges money
• creates records
• sends messages
• mutates state

…I start with parallel tool use disabled.

Then I add idempotency keys before I get fancy.

Example: idempotency around a charge tool

If your agent can trigger billing, don’t trust the model to "probably only call once."

Wrap the side effect.

import crypto from "node:crypto";

type ChargeArgs = {
  customerId: string;
  amountCents: number;
  invoiceId: string;
};

function buildIdempotencyKey(args: ChargeArgs) {
  return crypto
    .createHash("sha256")
    .update(`${args.customerId}:${args.invoiceId}:${args.amountCents}`)
    .digest("hex");
}

async function chargeCustomer(args: ChargeArgs) {
  const idempotencyKey = buildIdempotencyKey(args);

  return fetch("https://api.example.com/charge", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "Idempotency-Key": idempotencyKey
    },
    body: JSON.stringify(args)
  });
}

If the model retries the same action, your system should make the duplicate harmless.

That is what production-safe looks like.

The real bug factory is the round trip

The production path is not just “model emits tool call.”

It’s this:

1. Claude or GPT-4o emits a tool call
2. your app executes it
3. your app sends the tool result back
4. the model continues based on that result

That handoff is where a lot of systems quietly break.

This is where I now inject fixtures for:

• malformed tool inputs
• partial API failures
• duplicate execution attempts
• missing tool_result payloads
• slow responses and timeouts
• successful call with semantically wrong data

Most teams test the first request.
A lot of the breakage lives in the second.

My minimum viable agent QA checklist

You do not need a giant eval platform on day one.

But if the agent can hit real systems, you need a small, ruthless replay suite.

1. Schema fixture

Test every high-value tool against strict JSON Schema rules.

If you use OpenAI Structured Outputs, malformed args should be rare enough to treat as failures.

2. Wrong-tool fixture

Give the agent a prompt where two tools look plausible.

Make sure Claude or GPT-4o picks the right tool based on business rules, not keyword proximity.

3. Duplicate-action fixture

Replay the same tool call twice.

Verify your app blocks it, deduplicates it, or safely replays it.

4. Partial-failure fixture

Simulate a timeout, a 500, or a partial payload from an HTTP Request Tool in n8n.

Confirm the agent does not continue with corrupted assumptions.

5. Round-trip fixture

Test the full flow:

tool_use -> execution -> tool_result -> follow-up reasoning

A lot of “agent bugs” are really handoff bugs.

6. State fixture

Seed stale memory and verify the agent does not drag old context into a new job.

If you use n8n memory features like Chat Memory Manager, test memory boundaries explicitly.

7. Regression dataset

Save every interesting production failure as a replayable case.

Not a screenshot.
Not a Slack thread.
A dataset.

What I actually compare in practice

Approach	What it’s best for
OpenAI Structured Outputs	Schema adherence with strict: true, malformed-JSON regression tests, argument validation
Anthropic Tool Use	Duplicate actions, orchestration flow, multiple tool_use blocks, tool_result handoff testing
LangSmith Evaluations	Datasets, tracing, offline regression testing, backtesting against production traces

That table is boring.
Good.
Boring is what keeps agents from embarrassing you.

A simple replay harness beats vibes-based QA

Even a tiny local replay runner is better than “I tried three prompts and it seemed fine.”

Here’s a minimal example in Python:

import json
from typing import Callable

fixtures = [
    {
        "name": "duplicate_charge_attempt",
        "prompt": "Charge invoice INV-1001 for customer CUST-9",
        "expected_tool": "charge_customer",
        "expected_once": True,
    },
    {
        "name": "wrong_tool_ambiguity",
        "prompt": "Update the CRM note but do not send an email",
        "expected_tool": "update_crm_note",
        "expected_once": True,
    },
]


def run_fixture(agent: Callable, fixture: dict):
    result = agent(fixture["prompt"])

    assert result["tool_name"] == fixture["expected_tool"], \
        f"wrong tool: {result['tool_name']} != {fixture['expected_tool']}"

    if fixture.get("expected_once"):
        assert result["tool_call_count"] == 1, \
            f"duplicate tool call: {result['tool_call_count']}"

    return {"fixture": fixture["name"], "status": "pass"}


for fixture in fixtures:
    print(json.dumps(run_fixture(agent, fixture)))

This is not fancy.
It doesn’t need to be.

The point is to make failures replayable.

You probably don’t need a huge eval stack yet

If you’re building a narrow internal automation in n8n that:

• reads a Google Sheet
• calls an HTTP endpoint
• posts to Discord

…you may not need a heavyweight evaluation setup.

A curated replay suite plus tracing is often enough.

But “small” does not mean “casual.”
It means fewer fixtures, chosen carefully.

LangSmith’s guidance here is solid: break the system into critical components like tool invocation and output formatting, start with 5–10 curated examples, and use offline evaluation for regression testing.

That is just software testing with better nouns.

The compute-cost problem nobody likes to admit

There’s another reason teams under-test agents: regression runs cost money.

If every replay suite feels like it’s burning tokens in real time, teams become weirdly comfortable with less testing than they should be.

That’s one reason predictable compute matters for agent engineering.

When you’re running lots of evals, retries, prompt iterations, and workflow replays, per-token billing changes behavior. Teams test less. They avoid broad backtests. They skip useful fixtures because they don’t want surprise bills.

That’s exactly the pricing model I’ve grown to hate for serious automation work.

If you’re building agents that run all day in n8n, Make, Zapier, OpenClaw, or custom workflows, flat-rate API access is just easier to work with. You can actually afford to be disciplined.

Standard Compute is interesting for that reason: it’s a drop-in OpenAI-compatible API with flat monthly pricing instead of per-token billing, and it routes across models like GPT-5.4, Claude Opus 4.6, and Grok 4.20 behind the scenes.

That kind of setup makes a lot more sense for teams doing heavy replay testing and always-on automations than staring at token meters and pretending that won’t affect QA.

The opinion I wish someone had forced on me earlier

If your agent can call real APIs, it is not a chatbot with extra steps.
It is an integration surface.

Treating agent QA like prompt craftsmanship is how you get:

• beautiful demos
• haunted production logs
• duplicate side effects
• expensive mistakes

Treating it like integration testing is how you sleep.

The weird part is that better models make discipline more necessary, not less.

Once GPT-4o gives you strict schema adherence and Claude gives you cleaner tool orchestration, the remaining failures stop being mysterious.

They become your responsibility.

That’s actually good news.

“The model is weird” is hard to fix.
A replayable fixture for malformed JSON, duplicate actions, partial API failures, and stale memory is fixable tomorrow morning.

That’s the takeaway I wish I had on day one:

Don’t ask whether your agent is smart.
Ask whether it passes the same ugly cases twice in a row.

That’s when it’s ready.