Originally published at kunalganglani.com — read it there for inline code, hero image, and live links.

How to Do Agent Tool Call Failure Testing [2026 CI Harness]

By the end of this, you’ll have agent tool call failure testing running in CI: a local, deterministic tool stub that injects timeouts, 429s, partial failures, and malformed responses. You’ll also have a golden trace workflow so you can diff “how the agent behaved” across runs and catch regressions without an LLM judge.

Here’s the part most teams learn the expensive way. Production agent incidents usually aren’t “the model can’t reason.” They’re the unglamorous stuff: a tool timed out, a retry doubled a side effect, a rate limit hit mid-loop, or an HTTP 500 got treated like success because nobody bothered to classify errors.

When I built this site’s multi-agent publishing pipeline (261+ posts shipped through a 7-agent system), the failures that burned time weren’t clever prompt issues. They were tool-loop reliability problems. A deterministic gate caught more real problems than “upgrade the reviewer model” ever did. That’s the posture for this post. Stop debating eval theology and start breaking your tools on purpose.

I’m going to stay narrow: tool-call failure modes and deterministic assertions. If you want broader evals (task success, rubric scoring, etc.), bolt that on later via my AI in production and AI agents pillar hubs.

What is agent tool call failure testing (and how is it different from general agent eval)?

Agent tool call failure testing is when you deliberately inject tool failures (timeouts, rate limits, partial side effects, malformed outputs) into agent runs and assert the agent recovers safely and deterministically.

That’s not “general agent evaluation.”

General evals ask: “Did the agent solve the task?” Tool failure testing asks: “When reality gets messy, did the agent make things worse?” Different question. Different tooling.

The distinction I use:

• General agent eval: correctness, helpfulness, format, latency, cost, groundedness. Often dataset-driven, sometimes LLM-judge-driven.
• Tool-call failure testing: retries, backoff, idempotency, stop conditions, loop detection, error taxonomy, trace integrity. This should be mostly deterministic and CI-friendly.

Why it matters now: the agent stack keeps growing. Tool calling providers. MCP-style tool servers. Orchestrators. More network edges than your demo environment, and way more ways to hit timeouts and 429s.

If you’re building AI agents, you want a harness that answers “what happens when tools are unreliable?” before users answer it for you.

Why not just use an LLM judge?

[YOUTUBE:h8gMhXYAv1k|What is Tool Calling? Connecting LLMs to Your Data]

Because an LLM judge is the wrong instrument for this job.

For tool reliability, you want checks that are:

1. Structurally decidable (you can prove them from the trace)
2. Deterministic (CI should not be a coin flip)
3. Actionable (it points to the exact step that broke)

This is the core argument Ashwin Ugale makes with tracelint: a whole category of agent bugs can be caught by inspecting execution traces. Schema-invalid tool arguments. Continuing after tool errors. Stuck loops with identical tool calls. He even proposes CI exit codes like 0 (clean), 2 (structural defect), 3 (input error).

I agree with the philosophy.

Where teams mess this up is they treat trace linting as something you do after the fact. “Let’s lint production traces.” That’s useful, but it’s reactive.

You want to generate the failures on demand. That’s how you get regression tests instead of postmortems.

If you’re already running broader eval gates, see my agent orchestration and AI in production posts. This one is narrower. It’s a chaos harness for tools.

What it catches (a failure-mode taxonomy you can actually test)

Most tool-call failures fall into a handful of buckets. You don’t need 50 tests. You need about a dozen that cover the scary combinations.

Here’s a taxonomy I’ve actually found useful, plus what I’d assert. I’m going to keep the numbers concrete because vague test plans don’t ship.

Network and transport failures

• Hard timeout: client waits 2,000 ms, then aborts.
  • Assert: agent either retries with a budget, or escalates, but never proceeds as if the tool succeeded.
• Connect/DNS failure: tool host unreachable.
  • Assert: same as timeout. Also assert the agent doesn’t spin in a tight loop.
• Slow streaming / partial response: tool starts returning bytes, then stalls.
  • Assert: agent cancels and retries safely (or aborts).

HTTP semantics failures (when tools are HTTP APIs)

HTTP behavior isn’t folklore. It’s a spec.

• 429 Too Many Requests: rate limiting.
  • RFC 6585 defines HTTP 429 explicitly as “Too Many Requests” for rate limiting (Mark Nottingham, Roy T. Fielding).
  • Assert: agent respects Retry-After if present, adds jitter, and doesn’t burn the entire retry budget instantly.
• 5xx: server errors.
  • Assert: retry with exponential backoff, but cap attempts.
• 4xx: client errors (schema drift, auth, forbidden).
  • Assert: usually do not retry. Surface a crisp error.

If you want to be precise about retries, anchor on HTTP semantics rather than vibes. RFC 9110 is the canonical spec for core HTTP semantics (Julian Reschke, Roy T. Fielding, Mark Nottingham).

Tool protocol failures

• Malformed JSON in tool output
• Schema drift (missing field, type mismatch)
• Truncated pagination (agent assumes it saw “everything”)

Assertions here are clean: parsing must succeed, schema must validate, pagination must be explicit.

Partial failures (the ones that actually burn you)

This is the gap in a lot of agent-evals discourse.

A partial failure is when the tool succeeded server-side but the agent never received (or never processed) the response. The classic example is POST /charge succeeds, but the client times out and retries. Congratulations, you charged twice.

You should test at least two partial failure types:

1. Side effect applied + response lost (timeout after commit)
2. Side effect not applied + response lost (timeout before commit)

The correct behavior is different. Your agent needs idempotency keys and a way to query state.

If you ship anything with side effects, read that again. This is where “retry logic” quietly turns into “fraud logic.”

Stuck loops and duplicate calls

• Same tool called 5 times with identical arguments.
• Same tool called repeatedly after a terminal error.

Yes, you can catch these via trace linting (like Ashwin’s examples). It’s stronger when you can force the agent into the corner case in a repeatable harness and then assert it breaks out.

Try it: a drop-in adversarial tool stub + seedable chaos

The boring answer is the right one. Do not run chaos tests against real third-party APIs in CI. You’ll get flaky builds and surprise bills.

Instead, run a local “tool server” stub where every endpoint can be scripted to fail in deterministic ways.

You can do this in any language. I’ll describe the architecture in a framework-agnostic way, and then I’ll give you code that’s actually runnable.

The harness design (three pieces)

1. Scenario script: a list of tool calls and faults you want injected.
2. Fault-injecting tool server: local HTTP server implementing your tools.
3. Trace recorder + assertions: record tool calls, responses, and agent decisions. Then validate them.

This is “chaos engineering, but for tool calls.” AWS describes fault injection as disruptive experiments that stress workloads to observe behavior and improve resiliency, with templates and guardrails like stop conditions (AWS Fault Injection Service documentation).

You don’t need AWS FIS for agent tools. You do need the discipline. Templates + stop conditions + pre-prod.

A minimal local tool server with scripted faults

Below is a compact Node.js server (Express) that:

• Implements charge_card and get_charge_status
• Supports a deterministic scenarioId
• Can inject: timeout, 429 with Retry-After, 500, malformed JSON, and “success-but-response-lost” partial failure

// toolstub.js
import express from 'express';
import bodyParser from 'body-parser';

const app = express();
app.use(bodyParser.json());

// In-memory ledger so we can assert side effects deterministically.
// chargeId -> { amountCents, idempotencyKey, status }
const ledger = new Map();

// Scenario scripts define what happens on the Nth call.
// Deterministic: scenarioId + step index.
const scenarios = {
  "timeout-then-success": [
    { fault: "timeout", ms: 2500 },
    { fault: "success" }
  ],
  "429-respect-retry-after": [
    { fault: "429", retryAfterSec: 2 },
    { fault: "success" }
  ],
  "partial-success-lost-response": [
    { fault: "partial_success_lost_response" },
    { fault: "success" }
  ],
  "malformed-json": [
    { fault: "malformed_json" }
  ]
};

function getStep(req) {
  const scenarioId = req.header('x-scenario-id') || 'default';
  const step = Number(req.header('x-scenario-step') || '0');
  return { scenarioId, step };
}

function sleep(ms) {
  return new Promise(resolve => setTimeout(resolve, ms));
}

app.post('/tools/charge_card', async (req, res) => {
  const { scenarioId, step } = getStep(req);
  const script = scenarios[scenarioId] || [{ fault: "success" }];
  const action = script[Math.min(step, script.length - 1)];

  const { amountCents, idempotencyKey } = req.body;
  if (!amountCents || !idempotencyKey) {
    return res.status(400).json({ error: 'missing amountCents or idempotencyKey' });
  }

  if (action.fault === 'timeout') {
    await sleep(action.ms);
    // Simulate client timeout by never responding.
    return;
  }

  if (action.fault === '429') {
    res.set('Retry-After', String(action.retryAfterSec));
    return res.status(429).json({ error: 'rate_limited' });
  }

  if (action.fault === 'malformed_json') {
    res.set('Content-Type', 'application/json');
    return res.status(200).send('{"ok": true,'); // invalid JSON
  }

  // Implement idempotency: same idempotencyKey -> same chargeId.
  let existing = [...ledger.entries()].find(([, v]) => v.idempotencyKey === idempotencyKey);
  if (existing) {
    const [chargeId, v] = existing;
    return res.status(200).json({ chargeId, status: v.status, deduped: true });
  }

  const chargeId = `ch_${ledger.size + 1}`;
  ledger.set(chargeId, { amountCents, idempotencyKey, status: 'succeeded' });

  if (action.fault === 'partial_success_lost_response') {
    // Side effect applied, but agent never sees response.
    await sleep(2500);
    return;
  }

  return res.status(200).json({ chargeId, status: 'succeeded', deduped: false });
});

app.get('/tools/get_charge_status', (req, res) => {
  const chargeId = req.query.chargeId;
  const entry = ledger.get(chargeId);
  if (!entry) return res.status(404).json({ error: 'not_found' });
  return res.status(200).json({ chargeId, status: entry.status });
});

app.get('/__ledger', (req, res) => {
  // Test-only endpoint to assert “no double charge”.
  return res.status(200).json({
    charges: [...ledger.entries()].map(([chargeId, v]) => ({ chargeId, ...v }))
  });
});

app.listen(7071, () => {
  console.log('Tool stub listening on http://localhost:7071');
});

This is intentionally simple. In a real setup, I’d add:

• Per-tool latency distributions
• DNS failures (simulate at the client)
• Random-but-seeded fault schedules
• Stricter schema validation on both request and response

Determinism rules (so CI doesn’t rot into noise)

If your harness is nondeterministic, you’ll stop trusting it within two weeks. Every team does.

My rules:

• Every run has a scenarioId and a seed.
• Every tool call increments scenarioStep.
• Timeouts are simulated by “hang and let the client timeout,” not by random sleeps.
• Never hit the real network in CI. Stub everything.

If you’re already doing agent testing, wire this into your broader production AI pipeline.

How do you simulate tool-call timeouts deterministically without flakiness?

Timeout tests get flaky when you rely on actual scheduling jitter.

Make the boundary explicit:

• Your tool client sets a hard timeout (say 1,500 ms).
• Your stub tool server intentionally hangs for 2,500 ms.
• Your assertion checks agent behavior after a timeout error.

That’s deterministic because it doesn’t depend on whether your CI runners are “slow today.”

Two things matter:

1. The agent must treat timeout as unknown state (not failure, not success).
2. The agent must have a retry budget (count-based and/or time-based).

I like budgets that combine both:

• Max attempts: 3
• Max total elapsed on a tool: 10 seconds
• Max total tool calls per task: 20 (stop condition)

Those numbers aren’t holy. They’re defaults you can defend in a design review.

If you want a deeper dive into latency budgets, I’ve written about it in AI agent latency budgets and LLM latency benchmark methodology.

How do you test partial failures where the tool applied a side effect but the agent didn’t receive the response?

Partial failures are where you find out if you’re doing distributed systems or just roleplaying.

The harness needs two features:

1. A side-effect ledger you can query (/__ledger in the stub above)
2. Replay capability: the agent retries the exact tool call

The scenario to test

• First charge_card call applies a charge and then hangs (response lost).
• Agent times out at 1,500 ms.
• Agent retries.

Now assert:

• Ledger contains exactly 1 charge for that idempotency key.
• Agent did not claim success before confirmation.
• Agent reconciles by calling get_charge_status (or by reusing the idempotent response).

That’s the difference between “we retried” and “we caused a double spend.”

If you’ve built microservices with payment flows, this will feel familiar. If you haven’t, this is where agent projects learn distributed systems the hard way.

How do you design and test idempotency keys for agent tools?

Idempotency is not optional for tools with side effects. If your agent can call a tool twice, your tool must be safe to call twice.

A practical idempotency key design

I want keys that are:

• Deterministic for the user intent
• Unique enough to avoid collisions
• Scoped to a tool + operation

A pattern that works:

{tool}:{operation}:{userRequestId}:{normalizedArgsHash}

Example:

payments:charge:rq_8f3a:sha256(amount=1299|currency=USD|customer=123)

Two tests you should add:

1. Replay-safe retry test: same idempotency key, same args. Must not create a second side effect.
2. Collision test: different args but same idempotency key. Must be rejected with a 409 (or a tool-specific error) instead of silently doing the wrong thing.

And then the one most teams skip:

1. Timeout-after-commit test: tool commits the side effect, response is lost. Retry must return the original result.

This isn’t “just payments.” It’s also:

• ticket creation
• email sending
• database writes
• file deletion
• workflow approvals

Anything with side effects.

How should agents respond to HTTP 429 rate limits?

If your agent ignores 429s, it will DoS your own tools. Or someone else’s. Either way, you’re going to have a bad week.

RFC 6585 defines 429 Too Many Requests (Mark Nottingham, Roy T. Fielding). In practice, many APIs pair 429 with Retry-After. Treat that header like a contract.

My policy defaults

• If Retry-After is present, wait that duration.
• Add jitter of ±20% so fleets don’t thundering-herd.
• Use exponential backoff when Retry-After is absent: 250ms, 500ms, 1s, 2s.
• Hard cap retries at 3 (per tool call) and 10 total tool calls for the task.
• Track a per-tool retry budget so one failing tool doesn’t consume the entire run.

How to test it

Create a scenario:

• First call returns 429 with Retry-After: 2.
• Second call succeeds.

Assertions:

• The second call occurs >= 2 seconds after the first.
• The agent does not make more than 1 call during the Retry-After window.
• The run completes without exceeding budgets.

If you’re measuring LLM cost in production, 429 handling is also a cost control mechanism. Retries multiply tool calls and tokens.

How do you detect and fail stuck retry loops or repeated identical tool calls?

This is where trace-based linting earns its keep.

You can detect stuck loops without any model judgment:

• Same tool name + same normalized args repeated N times
• No change in observation payloads
• No new user input

Pick an N and enforce it. I use 5 as the default because I’ve never seen a good reason to call the same failing tool 12 times in a row.

This lines up with the “structurally decidable defects” argument from Ashwin Ugale. The extra step I’m pushing here is: force the loop using fault injection, then assert the agent breaks out.

If you want to go deeper on loop shapes, read The cracks in the loop. Most agent loops still look like while(true) with vibes. That’s why these bugs exist.

Golden traces: capture, normalize, diff

Task-success tests are fragile. Golden traces are weirdly robust.

A golden trace is a canonical record of:

• tool calls (name, args, timestamps)
• tool results (status, payload)
• agent decisions (retry, abort, escalate)

Then in CI, you rerun the scenario and compare the trace.

What you should diff (and what you should ignore)

Diff:

• tool call sequence
• number of attempts
• backoff delays (within tolerance)
• stop condition trigger points
• final outcome classification (success / aborted / needs-human)

Ignore:

• random IDs
• timestamps (normalize to offsets)
• model tokens
• non-deterministic wording in final natural-language output

The goal isn’t “the agent said the exact same sentence.” The goal is “the agent took the same safe actions.”

This ties into the architectural framing that “the log is the state.” If you haven’t read it, start here: What if the log is the state?.

A simple golden trace format

Store JSON like:

• scenarioId
• seed
• events[] where each event is { type, toolName, argsHash, status, errorClass, tOffsetMs }

Then git diff becomes a surprisingly effective regression detector.

Stop conditions and guardrails (avoid runaway costs)

Tool chaos tests are supposed to be adversarial. Which means they can go off the rails.

Guardrails are not “enterprise process.” They’re how you prevent your agent from turning into a fork bomb.

AWS FIS calls out stop conditions as a first-class guardrail for fault injection experiments (AWS Fault Injection Service documentation). Steal that idea.

My baseline stop conditions:

• Max wall clock per scenario: 30 seconds in CI, 5 minutes in staging
• Max tool calls per run: 20
• Max retries per tool: 3
• Max spend per run: $0 in CI (stubbed). In staging, define a ceiling.

If you’re using real hosted models for these runs, your stop condition should include token budgets too. This ties directly into LLM cost and production AI hygiene.

For cost math on retries and tool calls, I’ve written a concrete model in Agent cost per task and Agent per-task cost calculation.

Metrics to log: reliability is observable or it’s imaginary

If you can’t measure it, you can’t improve it. And you can’t prove the harness helped.

Log these per tool:

• success rate (%), over N=100 runs (in staging)
• p50/p95/p99 latency (ms)
• retry count distribution
• 429 frequency
• timeout frequency
• “unknown outcome” count (partial failures)

Also log per run:

• total tool calls
• total elapsed time
• stop condition triggers

If you already have OpenTelemetry in place, you can map tool calls to spans. I’ve got an implementation-oriented guide in OpenTelemetry instrumentation for AI agents and a schema in AI agent observability logging schema.

And yes, observability is part of testing. If your trace doesn’t capture enough to debug, your test failure is just a new kind of outage.

The trade-offs nobody wants to admit

These are the three trade-offs I see teams dodge.

1) Determinism vs realism

CI wants determinism. Production has randomness.

So do both:

• CI: scripted faults, fixed seeds, golden trace diffs.
• Staging: randomized-but-seeded chaos runs with statistical thresholds.

2) Retrying vs asking for help

“Retry everything” is how agents cause damage.

You need a boundary:

• Retry: timeouts, 429 with Retry-After, transient 5xx
• Abort: schema invalid, auth failure, 4xx indicating bad request
• Ask user / HITL: ambiguous partial failure with side effects, or when budget is exhausted

If you want patterns for human approval, see tool approval patterns.

3) The log is your product

Most agent stacks treat logs as an afterthought. That’s backwards.

Your trace format becomes:

• your test artifact (golden traces)
• your incident artifact (postmortems)
• your governance artifact (who approved what)

I learned this building the site’s agent pipeline. Deterministic gates only work when the artifacts are stable.

When we rewrote slugs on live URLs once, we burned 907K impressions worth of link equity in one incident. That wasn’t “LLM weirdness.” It was an identity and state problem. Tool-call testing is the same class of problem. The artifact is the state.

CI integration: keep it fast

A practical setup:

1. Start the tool stub server as a background process.
2. Run a small suite: 6–12 scenarios.
3. Record traces to ./artifacts/traces/.
4. Run deterministic linting rules.
5. Diff against golden traces.

If you’re building a broader gate, tie it into AI engineering evals and agent evaluation harness.

Also: don’t run this on every unit-test shard. Run it once per PR. It’s a system test.

A quick reference table: failure → expected agent behavior

Use this as your starting policy. You’ll tune it.

Failure mode	Example signal	Retry?	Backoff	Must use idempotency?	When to abort	Trace assertion
Timeout (no response)	client timeout at 1500ms	Yes (<=3)	exp (250ms→2s)	Yes for side effects	after budget	never claim success without confirmation
429 rate limit	HTTP 429 + Retry-After: 2	Yes (<=3)	wait Retry-After + jitter	Not required	after budget	next call >= Retry-After
5xx	HTTP 500/503	Yes (<=3)	exp + jitter	Yes for side effects	after budget	retries capped, no loop
4xx bad request	HTTP 400	No	none	N/A	immediate	surface tool error
Auth failure	HTTP 401/403	No	none	N/A	immediate	ask for re-auth/HITL
Malformed JSON	parse error	No (usually)	none	N/A	immediate	fail run with structural defect
Partial success + lost response	ledger shows write, client timed out	Conditional	cautious	Yes	if cannot reconcile	assert no duplicate side effects
Pagination truncation	missing next cursor	Conditional	none	N/A	if incomplete	agent must request next page
Stuck loop	same tool+args repeated 5x	No	none	N/A	stop condition	fail CI with loop defect

Honest limitations

This harness won’t solve everything.

• It won’t tell you if your agent’s reasoning is “good.” It tells you if your agent is safe under tool failure.
• Golden traces can get noisy if your agent’s planner is nondeterministic. You may need to normalize or pin planner settings.
• Some providers hide tool-call internals. If you can’t capture traces, you’re flying blind.

The hardest part is cultural.

Teams love shipping new tools. They hate hardening the boring ones. Reliability work isn’t sexy, but it’s what keeps your agent from becoming a liability.

If you want the security angle, pair this with prompt injection regression tests and the AI security leader playbook. Reliability and security failures look identical to users. They just see “the agent did something wrong.”

Here’s my prediction for 2026. The teams that win with agentic AI won’t be the ones with the fanciest prompts. They’ll be the ones who treat tool calls like a production distributed system. Build the harness now. Then watch everyone else scramble to copy you six months later.

---

Originally published on kunalganglani.com