My refund agent looked smart until it lied about Shopify, Stripe, and Zendesk 3 different ways

I thought I had a prompt problem.

I didn’t.

I had a state problem.

My support agent could read a Zendesk ticket, find the Shopify order, issue a Stripe refund, update the ticket, and reply to the customer. In staging, it looked great. In production, it started doing something much worse than "being wrong."

It was confidently telling customers their refund was complete when the workflow had only fired an API call and hoped for the best.

That distinction matters.

Because once an LLM can call real systems like Shopify, Stripe, and Zendesk, the failure mode changes. The model doesn’t need to hallucinate a tool name to hurt you. It just needs to overstate what happened after a valid tool call.

That’s the trap.

The real bug: valid tool calls, invalid conclusions

A lot of teams hit an "agent failed a task" moment and immediately do one of these:

• tighten the prompt
• add more warnings
• enforce stricter JSON schemas
• switch models
• add more examples

Sometimes that helps.

But if your agent can already call the right API with the right arguments, prompt work stops being the main lever.

OpenAI’s Structured Outputs are a real improvement here. Their published evals showed gpt-4o-2024-08-06 hitting 100% schema adherence on complex JSON schemas, versus less than 40% for gpt-4-0613.

That’s great.

But schema adherence is not operational truth.

You can have perfect JSON and still ship a support workflow that:

• reports success before money moved
• retries without preserving certainty
• drifts out of sync with async systems

That’s what bit me.

1) Shopify: a Refund object is not proof that money moved

This was my first clue.

The agent called Shopify refundCreate, got back a Refund object, and then told the customer:

Your refund has been processed.

Looks reasonable, right?

Not really.

Shopify’s docs are explicit: the existence of a Refund object does not guarantee the financial transaction completed. The actual outcome lives on the related OrderTransaction objects, which can be pending, processing, success, or failure.

So this pattern is broken:

1. call refundCreate
2. see refund.id
3. tell the customer it’s done

That’s not confirmation. That’s optimism with JSON.

The mutation is not the problem

mutation RefundOrder @idempotent(key: "refund-order-123") {
  refundCreate(input: {
    orderId: "gid://shopify/Order/123",
    note: "Customer requested partial refund"
  }) {
    refund { id }
    userErrors { field message }
  }
}

The bug happens after the mutation.

You need a second step that checks the transaction state before customer-facing messaging.

Safer pattern

async function refundInShopify(orderId: string) {
  const refund = await shopify.refundCreate({ orderId });

  if (refund.userErrors?.length) {
    return { status: "failed", reason: refund.userErrors };
  }

  const txns = await shopify.getOrderTransactions(orderId);
  const refundTxn = txns.find(t => t.kind === "refund");

  if (!refundTxn) {
    return { status: "unknown", reason: "No refund transaction found" };
  }

  switch (refundTxn.status) {
    case "success":
      return { status: "completed", refundId: refund.refund.id };
    case "pending":
    case "processing":
      return { status: "pending", refundId: refund.refund.id };
    case "failure":
      return { status: "failed", refundId: refund.refund.id };
    default:
      return { status: "unknown", refundId: refund.refund.id };
  }
}

That one extra verification step changes the customer message from guesswork into something defensible.

2) Stripe: timeouts turn "retry" into a reliability bug

Stripe is where this gets dangerous.

Imagine this flow:

1. your agent sends POST /v1/refunds
2. network hiccup
3. worker times out
4. response never gets persisted
5. model decides to "try again"

Now you have uncertainty.

Did Stripe create the refund?
Did it fail?
Did it succeed and you just lost the response?

This is exactly why Stripe idempotency keys exist.

And too many agent workflows still treat idempotency as optional.

It’s not optional.

It’s the thing that lets you retry without destroying your chain of evidence.

Correct Stripe call

curl https://api.stripe.com/v1/refunds \
  -u "sk_test_...:" \
  -H "Idempotency-Key: 8b5b9f2e-6f8d-4f3d-a6d8-2f0f4d7f9c21" \
  -d charge=ch_123

Stripe stores the first result for a given idempotency key and returns the same status code and body on retries, including 500 errors. If you reuse the same key with different parameters, Stripe rejects it.

That means your workflow needs to persist the key before the call, not after.

Bad retry logic

async function badRefundRetry(chargeId: string) {
  try {
    return await stripe.refunds.create({ charge: chargeId });
  } catch {
    // terrible: new request identity, no certainty
    return await stripe.refunds.create({ charge: chargeId });
  }
}

Better retry logic

import { randomUUID } from "crypto";

async function createRefundWithRecovery(chargeId: string, existingKey?: string) {
  const idempotencyKey = existingKey ?? randomUUID();

  await db.refundAttempts.upsert({
    chargeId,
    idempotencyKey,
    status: "started"
  });

  try {
    const refund = await stripe.refunds.create(
      { charge: chargeId },
      { idempotencyKey }
    );

    await db.refundAttempts.update({
      chargeId,
      idempotencyKey,
      status: "completed",
      refundId: refund.id,
      rawResponse: JSON.stringify(refund)
    });

    return refund;
  } catch (err) {
    await db.refundAttempts.update({
      chargeId,
      idempotencyKey,
      status: "unknown",
      error: String(err)
    });

    throw err;
  }
}

That unknown state matters.

A lot of teams try to avoid it because it feels messy. But "unknown" is honest. Telling the model to guess is not.

3) Zendesk: async jobs and rate limits expose toy workflows fast

Zendesk is where polished demos usually fall apart.

Two reasons.

Rate limits are not a suggestion

Zendesk Support and Help Center API limits vary by plan. Responses can include headers like:

X-Rate-Limit: 700
X-Rate-Limit-Remaining: 699

If you hit the limit, you can get 429 Too Many Requests plus Retry-After.

A model that keeps hammering the API because it wants to be helpful is not helping.

It’s just an expensive loop.

Some actions are jobs, not immediate completion

Bulk ticket updates are a classic example.

The workflow sends the update, gets an acknowledgment, and assumes the tickets changed.

But Zendesk job statuses can sit in:

• queued
• working
• failed
• completed

If your agent refunds 40 orders and bulk-updates 40 tickets, you cannot assume the ticket side finished just because the first call returned 200 or 202.

You need to poll the job URL and reconcile failures.

Minimal polling example

async function waitForZendeskJob(jobStatusUrl: string) {
  for (let attempt = 0; attempt < 20; attempt++) {
    const job = await zendesk.get(jobStatusUrl);

    switch (job.status) {
      case "completed":
        return job;
      case "failed":
        throw new Error(`Zendesk job failed: ${job.id}`);
      case "queued":
      case "working":
        await sleep(3000);
        continue;
      default:
        throw new Error(`Unknown Zendesk job state: ${job.status}`);
    }
  }

  throw new Error("Zendesk job did not reach terminal state in time");
}

This is not glamorous engineering.

It is, however, the difference between a support agent that sounds polished and one that actually behaves reliably.

Prompt fixes vs orchestration fixes

This is the distinction I wish I had made earlier.

Approach	What it actually solves
Better prompts and tighter tool descriptions	Reduces bad reasoning and malformed calls before execution
Structured Outputs in GPT-4o or GPT-5	Improves schema adherence and valid arguments
Idempotency keys in Stripe and Shopify patterns	Prevents uncertainty and duplicate side effects during retries
Polling Shopify transactions and Zendesk jobs	Confirms real-world completion instead of assuming it
Durable state in LangGraph, n8n, or Make	Lets you recover after timeouts, 429s, crashes, or partial completion

This is why I think a lot of "LLM tool use reliability" discussions are framed too narrowly.

Once you touch money, tickets, orders, or customer records, you’re not debugging a chatbot anymore.

You’re doing distributed systems work.

The LLM is just one component in the chain.

What I trust now

If I had to compress this into one rule:

Never let the model communicate success from the first side-effecting response.

Here’s the cheat sheet.

| Check | Shopify refund flow | Stripe refund flow |
|----------|----------|
| Does the initial object guarantee money movement? | No. A Refund object alone is not proof the refund settled | Usually the refund response is the main record, but retries must preserve idempotency to keep certainty |
| Do you need post-call verification? | Yes. Check associated OrderTransaction status like pending, processing, success, or failure | Yes, especially after timeouts or network failures; verify using the same idempotent request history or follow-up retrieval |
| What should retries rely on? | Stored workflow state and explicit verification logic | The same idempotency key with the same parameters |

That last row is where a lot of agent workflows quietly fail.

Not because Shopify or Stripe are flaky.

Because the workflow was stateless during retries.

Do you need LangGraph, or are n8n / Make enough?

My opinion: not every support automation needs a full agent runtime.

If the flow is mostly deterministic:

• look up order
• issue refund
• update ticket
• send reply

Then n8n, Make, or even Zapier with explicit branches can be safer than an autonomous loop.

You can hard-code:

• retries
• wait steps
• rate limit handling
• persisted IDs
• human escalation branches

If the workflow is more open-ended and needs resumability across long windows, partial failure recovery, or branching investigation, then LangGraph starts making more sense.

Tiny LangGraph skeleton

from langgraph.graph import StateGraph, MessagesState, START, END

def mock_llm(state: MessagesState):
    return {"messages": [{"role": "ai", "content": "hello world"}]}

graph = StateGraph(MessagesState)
graph.add_node(mock_llm)
graph.add_edge(START, "mock_llm")
graph.add_edge("mock_llm", END)
graph = graph.compile()
graph.invoke({"messages": [{"role": "user", "content": "hi!"}]})

That example is trivial.

The important part is the mindset shift: state transitions, checkpoints, resumability, deterministic recovery.

Not "maybe GPT-5 or Claude Opus 4.6 will be more careful next time."

The pattern that finally stopped the lying

What actually fixed this for me was boring.

I split the workflow into phases:

1. prepare the action with validated inputs
2. execute with an idempotency key or equivalent request identity
3. persist external IDs and raw responses immediately
4. verify downstream state in Shopify, Stripe, or Zendesk
5. communicate only from verified state
6. escalate ambiguous or non-terminal cases to a human

That did more for reliability than all the prompt tuning combined.

If you’re running AI agents in production, cost pressure makes this worse

There’s one more thing people don’t talk about enough: per-token pricing pushes teams toward bad reliability decisions.

When every retry, poll, verification step, and recovery branch feels like metered spend, people start trimming the boring parts.

They skip verification.
They shorten retries.
They avoid durable state.
They let the model improvise because it looks cheaper in the moment.

That is exactly backwards.

Production-grade agent workflows need room for:

• retries
• polling
• recovery steps
• reconciliation passes
• long-running automations

That’s a big reason tools like Standard Compute are interesting to teams building support agents, n8n flows, Make scenarios, and custom automations. If you’re using the OpenAI-compatible API shape but want predictable flat-cost compute instead of per-token anxiety, it changes how aggressively you can design for reliability.

You stop asking, "Can we afford another verification pass?"

You start asking the better question:

"What would make this workflow stop lying to customers?"

That’s the right optimization target.

Final takeaway

If your refund agent sounds smart but sometimes lies, don’t assume the fix is a better prompt.

Check whether your workflow is doing any of these:

• confirming success from an initial API response
• retrying without idempotency
• failing to persist request identity
• skipping async job polling
• sending customer-facing messages before reconciliation

That’s usually where the real bug lives.

The painful lesson for me was simple:

Side effects are not chat turns.

They’re distributed transactions wearing a chatbot costume.

And once you treat them that way, your agents get a lot less charming and a lot more trustworthy.