100 Actions, 110 Side Effects: A Crash Test for AI Agent Tools

Ihar Ivaniuk — Thu, 08 Oct 2026 16:03:53 +0000

Your agent chose the right tool. The arguments were valid. The action was allowed.

Then the worker died.

On recovery, should it call the tool again?

A small crash test produced 110 side effects from 100 intended actions. Moving the local completion marker ahead of the call removed the duplicates – but a different crash left 10 actions missing.

All four implementations passed the happy-path control.

This is a reproducible tool-execution lab, with real worker termination and a simulated external provider. No LLM, real emails, payments, or customer systems are involved. That isolation is deliberate: even a perfect model cannot recover information that the execution layer never recorded.

The result that a successful demo hides

The lab runs 100 distinct logical operations per scenario. Every tenth operation gets one injected worker crash, followed by one retry in a fresh process.

There are two crash locations:

Before the effect: the provider receives the request, but the worker is killed before the provider applies it. For the “record before” strategy, the local marker already exists.
After the effect: the provider applies the action, then kills the worker before returning a receipt.

Here are the observed results. Each cell shows effects / missing operations:

Strategy	No crash	Crash before effect	Crash after effect
Retry without deduplication	100 / 0	100 / 0	110 / 0
Record completion after the call	100 / 0	100 / 0	110 / 0
Record completion before the call	100 / 0	90 / 10	100 / 0
Stable key, deduplicated by provider	100 / 0	100 / 0	100 / 0

These are constructed failures, not estimates of how frequently production systems fail. The 10 crashes are part of the test input.

The useful finding is the trade-off: a duplicate counter can show zero while your system silently loses work.

The two writes you cannot wish into one transaction

Consider this tool handler:

const receipt = await provider.execute(operation);
await operations.markCompleted(operation.id, receipt.id);

There are two separate changes: one at the provider, one in your application.

If the first succeeds and the worker dies before the second, your database still says the work is unfinished. Repeating it may create another external effect.

Swapping the lines changes the failure:

await operations.markCompleted(operation.id);
await provider.execute(operation);

Now a crash can leave a completed record for an action that never happened. A recovering worker sees the marker and skips the operation.

In the lab, that worker exits successfully. The external ledger is still short by ten actions.

The marker has confused “we intended to do it” with “we have evidence that it happened.”

An in_progress record is useful. Treating it as a completion receipt is the mistake. A lease can coordinate workers, but it does not establish whether a remote action happened before the lease holder disappeared.

What the fourth strategy actually changes

The last strategy gives each logical operation a stable identity and sends that identity to a provider that enforces deduplication.

The worker may attempt operation 40 twice. The provider recognizes the second attempt and returns the existing receipt instead of applying a second effect.

The important boundary is the one that owns the effect. Putting a cache around the caller leaves the gap between the remote write and the local record.

Amazon’s explanation of idempotent APIs describes both client request identifiers and the need to make token recording atomic with the service’s mutations. Our simulated provider performs its lookup and append in one synchronous parent-process callback. That is enough for this sequential experiment; a real service needs an appropriate durable, concurrent implementation.

A production request might carry this shape:

// Illustrative contract, not a production implementation.
await provider.execute(persistedOperation.payload, {
  idempotencyKey: persistedOperation.id,
});

“Persisted” matters twice:

The operation identity survives retries. A UUID generated once and stored is fine. A fresh UUID inside each attempt defeats deduplication.
The intended payload stays bound to that identity. A changed recipient, amount, or message is not automatically the same operation.

Do not use the payload alone as the identity. Two intentionally separate actions can have identical payloads. Conversely, reusing one operation ID for changed parameters should produce a conflict, not quietly execute a different action.

Stripe documents this distinction concretely: repeated keys replay stored results, mismatched parameters are rejected, and keys can be removed after they are at least 24 hours old. Its documentation also notes that stored results can include errors. Provider-specific rules and retention windows belong in your retry design.

A key is only useful when the receiver honors it.

Why this belongs in an agent discussion

The distributed-systems problem is older than agents. DEV already has articles about duplicate agent actions and job replay after a restart. This lab makes two crash boundaries directly comparable and includes lost work in the scorecard.

Agents add another place where an operation can be recreated. Your queue may retry a job while the planner independently decides to “try sending again.” If those paths mint different operation IDs, provider deduplication will treat them as different actions.

The application therefore needs a durable identity for the intended action across planner turns, tool calls, and worker attempts. Do not ask the model to invent that identity anew during recovery.

Checkpointing still helps. It just has a boundary: LangGraph’s Functional API documentation explicitly notes that a task that started but did not finish may run again. Completed task results can be restored; an unfinished external write still needs safe re-execution.

Human approval has a different job, too. Approval establishes that an action is allowed. It does not prevent an already-approved action from happening twice. Bind approval to the intended payload, then handle execution and recovery separately.

What if the provider has no idempotency contract?

First, check whether there is an authoritative way to retrieve the outcome using a client reference or provider operation ID.

After an ambiguous failure, a useful set of outcomes is:

Evidence	Next step
The provider confirms the effect	Store its receipt and complete locally
The provider authoritatively confirms no effect and no in-flight request	Retry under the applicable contract
The result is still uncertain	Keep it unresolved; reconcile or escalate

An empty search result is not necessarily proof that nothing happened. Indexing may lag, or the original request may still complete.

If neither deduplication nor reliable reconciliation exists, you have a product decision: accept the possibility of duplicates, accept the possibility of omission, or hold the operation for investigation. Make that trade-off visible. Do not label uncertainty as success to clear the queue.

An outbox can make the local intent and local business change atomic. Its dispatcher still needs a safe contract with the external destination. Moving the retry into another process does not close the remote commit gap.

Reproduce it

Save the complete script below as crash-lab.mjs, then run:

node crash-lab.mjs > results.json

The recorded run used Node.js 24.19.0 on Linux. It needs no packages or API keys. Run it in a local development environment: it creates temporary marker files and deliberately terminates only its own child workers with SIGKILL.

The parent process represents the provider and survives worker crashes. It records effects independently of the worker’s local markers. The script compares actual provider records against the 100 intended operation identities and asserts the expected outcomes.

The provider is an in-memory simulation. This experiment does not test provider or host crashes, concurrent requests, expired keys, changed payloads, network behavior, authorization, or LLM planning quality. Passing these cases is evidence about these cases, not a production-readiness certificate.

Full runnable crash lab

// Isolated educational experiment. No network, credentials, or real messages.
// Run: node crash-lab.mjs &gt; results.json
import { fork } from 'node:child_process';
import { existsSync, writeFileSync, mkdtempSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { fileURLToPath } from 'node:url';
import assert from 'node:assert/strict';

const self = fileURLToPath(import.meta.url);
const strategies = ['retry-only', 'record-after', 'record-before', 'provider-key'];

if (process.argv[2] === 'worker') {
  const [strategy, receipt, operation] = process.argv.slice(3);
  // A pre-written marker is deliberately treated as completion in record-before.
  if (strategy !== 'retry-only' &amp;&amp; existsSync(receipt)) {
    process.exit(0);
  }
  if (strategy === 'record-before') writeFileSync(receipt, 'claimed');
  process.on('message', (message) =&gt; {
    if (message.type !== 'ack') return;
    if (strategy !== 'retry-only') writeFileSync(receipt, message.providerId);
    process.disconnect();
  });
  process.send({ type: 'execute', operation,
    key: strategy === 'provider-key' ? operation : null });
} else {
  const directory = mkdtempSync(join(tmpdir(), 'agent-crash-lab-'));
  const rows = [];
  // Parent = simulated external provider. Worker = separate killable process.
  // Provider state survives worker death, but not parent/host death.
  async function run(strategy, boundary, injectFaults) {
    const effects = [];
    const receipts = new Map();
    let requests = 0;
    let kills = 0;
    let skippedWithoutEffect = 0;
    async function attempt(operation, receipt, shouldKill) {
      return new Promise((resolve, reject) =&gt; {
        const child = fork(self, ['worker', strategy, receipt, operation], {
          stdio: ['ignore', 'ignore', 'inherit', 'ipc'],
        });
        const timeout = setTimeout(() =&gt; {
          child.kill('SIGKILL');
          reject(new Error('Unexpected worker timeout'));
        }, 5000);
        child.on('error', reject);
        child.on('message', (message) =&gt; {
          if (message.type !== 'execute') return;
          requests++;
          if (shouldKill &amp;&amp; boundary === 'before-effect') {
            kills++;
            child.kill('SIGKILL');
            return;
          }
          // Atomic in this single parent event-loop callback. NOT a production DB.
          let providerId = message.key &amp;&amp; receipts.get(message.key);
          if (!providerId) {
            providerId = `effect-${effects.length + 1}`;
            effects.push({ operation: message.operation, providerId });
            if (message.key) receipts.set(message.key, providerId);
          }
          if (shouldKill &amp;&amp; boundary === 'after-effect') {
            kills++;
            child.kill('SIGKILL'); // effect committed; worker gets no receipt
            return;
          }
          child.send({ type: 'ack', providerId });
        });
        child.on('exit', (code, signal) =&gt; {
          clearTimeout(timeout);
          if (signal === 'SIGKILL' &amp;&amp; shouldKill) resolve(false);
          else if (code === 0) resolve(true);
          else reject(new Error(`Unexpected worker exit: ${code}/${signal}`));
        });
      });
    }
    for (let index = 1; index &lt;= 100; index++) {
      const operation = `tenant-demo:notification:${index}`;
      const receipt = join(directory, `${strategy}-${boundary}-${injectFaults}-${index}`);
      const ok = await attempt(operation, receipt, injectFaults &amp;&amp; index % 10 === 0);
      if (!ok) assert.equal(await attempt(operation, receipt, false), true);
      if (!effects.some((effect) =&gt; effect.operation === operation)) skippedWithoutEffect++;
    }
    const unique = new Set(effects.map((effect) =&gt; effect.operation)).size;
    return { strategy, boundary, injected: injectFaults, intended: 100,
      requests, kills, effects: effects.length, duplicateEffects: effects.length - unique,
      missingOperations: 100 - unique, skippedWithoutEffect };
  }
  try {
    // Happy-path controls: every implementation looks correct without a crash.
    for (const strategy of strategies) {
      const row = await run(strategy, 'none', false);
      assert.equal(row.effects, 100);
      assert.equal(row.missingOperations, 0);
      rows.push(row);
    }
    for (const boundary of ['before-effect', 'after-effect']) {
      for (const strategy of strategies) {
        const row = await run(strategy, boundary, true);
        assert.equal(row.kills, 10);
        const missing = boundary === 'before-effect' &amp;&amp; strategy === 'record-before' ? 10 : 0;
        const duplicates = boundary === 'after-effect' &amp;&amp; ['retry-only', 'record-after'].includes(strategy) ? 10 : 0;
        assert.equal(row.missingOperations, missing);
        assert.equal(row.duplicateEffects, duplicates);
        assert.equal(row.effects, 100 - missing + duplicates);
        rows.push(row);
      }
    }
    console.log(JSON.stringify({ node: process.version, platform: process.platform,
      experiment: '100 operations per row; kill every tenth initial attempt; one retry',
      scope: 'Single worker at a time; simulated external provider; no LLM; no network',
      assertions: 'passed', rows }, null, 2));
  } finally {
    rmSync(directory, { recursive: true, force: true });
  }
}

Bring these questions to your next demo

Ask the team to show one consequential operation and answer:

Where is its identity stored before execution starts?
Which system can prove the external effect happened?
What happens if the worker dies immediately before and immediately after that effect?
Does the test assert both no duplicates and no missing work?
What does the user see while the result is genuinely unknown?

Then run the two crash cases against a sandbox version of the real integration. Inspect the destination’s records, not just your worker’s logs.

Which tool in your system would be hardest to reconcile after a lost response – and what evidence would you trust?

Further reading: my software architecture review framework covers contracts, failure behavior, and operability beyond this one test.

What’s your take on Upwork in fall 2026?

Ihar Ivaniuk — Wed, 07 Oct 2026 23:37:09 +0000

Developers, what’s your take on Upwork in fall 2026? Are you finding solid projects at reasonable rates? Is it worth joining as an experienced engineer with no track record on the platform?

I’m a Full-Stack Engineer with 9+ years of experience in React/Next.js, Node.js, AI integrations, and automation. I’m considering Upwork as a way to find clients and would love to hear about your recent firsthand experience.

🖥️ Website: https://ihar-ivaniuk.com
💼 LinkedIn: https://www.linkedin.com/in/ihar-ivaniuk/

Should a portfolio simply showcase projects – or should it be a project itself?

Ihar Ivaniuk — Sat, 05 Sep 2026 11:11:56 +0000

I’m already in the middle of rebuilding ihar-ivaniuk.com — this time as a portfolio inspired by macOS and desktop interfaces.

Current version:
https://ihar-ivaniuk.com

Old version:
https://old.ihar-ivaniuk.com

The idea originally came from my girlfriend. Then I started seeing similar concepts all over Instagram, and eventually decided to build my own take on it.

Under the hood
Next.js 16
TypeScript
Motion
dnd-kit
next-intl
GitHub Actions
Vercel

Fellow developers, don’t scroll past ☝️☝️☝️
I’d genuinely love to hear what you think.

Was this redesign worth a couple dozen hours of work?

And of course, I’ll be happy to answer any questions in the comments 🙌

DEV Community: Ihar Ivaniuk