DEV Community: FetchSandbox

Why Paddle's subscription.activated arrives before subscription.created

FetchSandbox — Thu, 30 Jul 2026 13:58:08 +0000

You'd think events fire in the order things happen. They don't.

I was building a Paddle integration last week. Subscription billing, nothing fancy. Customer clicks buy, Paddle handles checkout, my app gets webhooks and updates the database.

The flow should be simple:

Customer completes checkout
subscription.created fires
subscription.activated fires
My app inserts a row on .created, updates status on .activated

That's what I built. It worked great in my head.

What actually happened

In production, subscription.activated arrived before subscription.created about 30% of the time.

My handler did a database insert on subscription.created:

case 'subscription.created':
  await db.insert(subscriptions).values({
    paddleId: event.data.id,
    status: 'created',
    customerId: event.data.customer_id,
  });
  break;

And an update on subscription.activated:

case 'subscription.activated':
  await db.update(subscriptions)
    .set({ status: 'active' })
    .where(eq(subscriptions.paddleId, event.data.id));
  break;

When .activated arrived first, the update found zero rows. No error, no exception. The WHERE clause just matched nothing. The update silently did nothing.

Then .created arrived and inserted the row with status created. But the .activated event was already gone. So the subscription was stuck in created status forever.

Customers had paid. Paddle showed them as active. My app showed them as pending. Support tickets started coming in.

Why this happens

Paddle does not guarantee webhook delivery order. Their docs mention it briefly but it's easy to miss when you're focused on the API endpoints.

The events are fired from different internal services. subscription.created comes from the subscription service. subscription.activated comes from the billing service after payment confirmation. They are async. They race.

This is not unique to Paddle either. Stripe has the same problem with payment_intent.created vs charge.succeeded. Most payment providers have some version of this.

The fix

The handler needs to be idempotent and order-independent. Every event should be able to create or update:

case 'subscription.created':
case 'subscription.activated':
  const status = event.event_type === 'subscription.activated' 
    ? 'active' : 'created';

  await db.insert(subscriptions)
    .values({
      paddleId: event.data.id,
      status,
      customerId: event.data.customer_id,
    })
    .onConflictDoUpdate({
      target: subscriptions.paddleId,
      set: { 
        status: sql`CASE WHEN ${status} = 'active' THEN 'active' ELSE ${subscriptions.status} END`
      },
    });
  break;

The key parts:

Both events can create the row if it doesn't exist
On conflict, active always wins over created regardless of arrival order
No silent failures, no missing updates

Testing this is the real problem

The ordering bug is easy to fix once you know about it. The hard part is reproducing it during development.

You can't control the order Paddle sends webhooks. You can't make .activated arrive first on demand. In testing you might run through the flow 20 times and the events always arrive in order. Then in production with real network latency and load, they don't.

I ended up testing this by sending the webhook events manually in the wrong order against a local sandbox. activated first, then created. Immediately saw the bug. Fixed it in 10 minutes.

The debugging in production took 4 hours.

If you're integrating Paddle or any payment provider with webhooks, test with events arriving in every possible order. Not just the happy path order from the docs.

Test Paddle webhook ordering in a sandbox →

The takeaway

Webhook events are not a queue. They are concurrent messages from different services that happen to be about the same thing. Your handler has to treat every event as potentially the first one it sees for that resource.

If your handler has an insert for one event type and an update for another, you have this bug. You just haven't hit it in production yet.

Webhook handlers that silently return 402: the event ordering bug most test suites miss

FetchSandbox — Tue, 21 Jul 2026 17:40:10 +0000

How to prove a handler is order-safe using permutation fuzzing

You're testing your Paddle webhook handler. You've got a test for subscription.created and subscription.activated, and maybe another one where you send them out of order. Looks good. All green. But weeks later, a customer hits a 402 on subscription.activated. No error in your logs. No alert. Just a failed charge and a confused user.

The problem isn't your test suite, it's the order of events.

A Paddle webhook handler that does an INSERT on subscription.created and an UPDATE on subscription.activated will silently return 402 when subscription.activated arrives first. The row doesn't exist yet. No error is raised. But if you send them in the "happy path" order, it works. Your test suite never sees the failure because it only runs one sequence. The bug is invisible until it hits production.

To prove a handler is order-safe, we built a permutation fuzzer in fetchsandbox MCP. It fires the same events in every permutation, including duplicates, and checks if the final state is the same in all of them. If it is, the handler is order-safe. If not, it returns the exact sequence that breaks it.

Here's what it found for the Paddle handler above:

When you send subscription.activated before subscription.created: the handler returns a 402.
When you send them in the "happy path" order: it returns a 200.
The fuzzer returns order_independent=False and shows you the diverging sequence.

After fixing the handler to use an upsert instead of separate INSERT and UPDATE, the fuzzer returns order_independent=True across every permutation and duplicate.

If you test API integrations in sandboxes constantly, you've hit this one too. I run it in Cursor every time I test a new webhook integration.

The fuzzer is in fetchsandbox MCP. It reads final state after each permutation, and reports order_independent=True only when every ordering converges to the same final state.

If you're trying to find webhook ordering bugs, or prove a handler is order-safe, this is the test you didn't know you needed.

Read the full spec for Paddle webhook testing at Paddle sandbox. Learn how fetchsandbox MCP works at fetchsandbox MCP.

We planted a Descope privilege-escalation bug — can your coding agent catch it?

FetchSandbox — Fri, 10 Jul 2026 18:39:39 +0000

We open-sourced a repo with a bug planted on purpose: github.com/fetchsandbox/playground.

The bug is in apps/descope — a small FastAPI "Agent Gateway" where AI agents exchange a Descope access key for a scoped session. The flaw: the exchange endpoint trusts whatever scopes the client requests. A read-only agent key can ask for users:write and get it.

This is the privilege-escalation shape that ships constantly in agentic auth. The token exchange returns 200, the happy path works, and nothing ever compares the granted scope against what the key was actually granted.

The 20-minute test drive

Two tasks, both run from your IDE (Cursor, Claude Code, or any MCP-capable editor). No Descope account, no API keys — the workflows run against FetchSandbox's hosted Descope sandbox over MCP, and each app ships a .mcp.json already wired.

Task 1 — greenfield. apps/descope-onboarding is a tiny notes app with placeholder auth. Ask your agent to add Descope OTP sign-up, but with one constraint: prove the OTP + session flow in the sandbox before writing any code, then propose the diff.

./fetchsandbox I'm adding Descope OTP sign-up to this app — prove the Descope
OTP + session flow in the sandbox before writing any code, then propose the
diff. I'll decide whether to apply.

Task 2 — brownfield. Point the agent at the Agent Gateway and ask it to audit:

./fetchsandbox our agent access-key exchange might be handing out more scope
than the key was granted — audit the descope agentic auth

What a real catch looks like

Static review is not the bar. "This looks vulnerable" is a guess. The bar is:

the escalation is reproduced — read-only key in, users:write session out
the proof shows buggy vs fixed on actual Descope routes (/v1/...)
there's a receipt URL — an openable run trace, not a claim in chat

Auth bugs live in the lifecycle: replayed magic links, expired sessions, JWTs that were decoded instead of verified, scope grants nobody re-checked. Mocks return clean 200s for all of these. That's the gap the playground is designed to expose.

The other planted bugs

If Descope isn't your stack, the repo also has broken Stripe (webhook dedup keyed on the wrong header — events processed 2–3×), Resend (bounces silently dropped, users stay "active"), Clerk (session validation skipped on one endpoint), and a few more. Each app is ~50–150 lines of Python.

The part where you tell us it fell flat

Findings go back as a PR — there's a FINDINGS_TEMPLATE.md in the repo. Paste your agent's session, the receipt URLs, and your honest reaction. Merged PRs show up on your GitHub contribution graph.

We explicitly want the failure reports: MCP wouldn't connect, the agent caught nothing, the proof felt fake. An honest "it didn't work" beats a polite green checkmark.

Start here: TESTING.md.

How to test Descope auth without real keys

FetchSandbox — Wed, 08 Jul 2026 14:04:28 +0000

Title options:

How to test Descope auth without real keys
How to test Descope passwordless auth without a project

Descope is easy to start with, but the part I would test first is not the OTP screen or the magic-link email.

It is what your app does with the session JWT after Descope returns it.

FetchSandbox has a Descope sandbox for the flows most apps wire up first:

email OTP signup
email magic-link sign-in
session refresh
agent/access-key exchange

No real Descope project. No real keys. No inbox setup.

The useful part is that the sandbox models the failure cases too: wrong OTP, expired session, replayed magic link, and the big one: accepting a forged session JWT because the app decoded it instead of verifying it.

Here is the concrete flow I would test.

Send a magic link:

POST /v1/auth/magiclink/signin/email

Verify the one-time token:

POST /v1/auth/magiclink/verify

The sandbox returns the shape your app should care about:

{
  "sessionJwt": "...",
  "refreshJwt": "...",
  "user": {
    "userId": "U2..."
  }
}

Call your protected route with the sessionJwt.

This is where the bug usually lives.

Bad handlers do something like:

const claims = jwt.decode(sessionJwt);
if (claims.role === "admin") allow();

That accepts whatever claims are inside the token. A forged alg: none token can become role=admin if the handler never checks the signature.

The fixed version verifies the session JWT against Descope's JWKS, or uses Descope's SDK validation path. It should reject alg: none, reject bad signatures, cache JWKS, and refresh on kid mismatch.

The sandbox route for this bug is meant to prove that difference:

forged unsigned token -> 401
valid signed session token -> 200

That is the whole point. Do not just test "magic link works." Test that the session you trust is actually verified.

You can also test refresh behavior:

POST /v1/auth/refresh
GET /v1/auth/me

If refresh returns 401, send the user back through sign-in. Do not keep rendering authenticated UI from a decoded-but-unverified token.

Docs are here if you want the exact Descope sandbox flows: https://fetchsandbox.com/docs/descope

Our CI mocked Stripe. Production still broke on step two.

FetchSandbox — Tue, 30 Jun 2026 14:23:58 +0000

Green CI. Broken staging. Sound familiar?

Our pipeline mocked POST /v1/payment_intents and asserted 200. Merge button enabled. First real checkout in staging failed because the handler never stored the PaymentIntent ID — step two of the webhook path read undefined.

Mocks proved shape. Not lifecycle.

What unit tests miss

CI check	What it proves	What it skips
Mocked `fetch()`	Request JSON looks right	Step 2 cannot read step 1's ID
Snapshot tests	Handler code unchanged	Webhook branch never runs
Manual smoke doc	Someone remembers sometimes	Agents ship faster than the checklist

The bug class: IDs do not chain across steps. Webhooks fire on mutations your mock never triggers.

What we run now — MCP locally, same workflow in CI

Before an agent touches payment code, we prove the provider flow in the IDE via FetchSandbox MCP:

quickrun stripe / accept_payment
→ create PaymentIntent
→ confirm / capture path
→ read back state

That is the contract the integration must satisfy — not a paragraph in the PR description.

Then CI runs the same curated workflow headless:

# .github/workflows/api-integration.yml
- name: Prove Stripe integration before deploy
  run: npx fetchsandbox run stripe --all --json

Exit code 0 → workflows passed, merge allowed.

Exit code 1 → named step failed (create_payment_intent, webhook reconcile, etc.) — PR blocked.

No staging dependency. No IP whitelist. No "works on my machine."

Why MCP belongs in the loop

Coding agents edit integration code constantly. Without a runnable proof:

The agent "fixes" the handler and moves on
CI mocks still pass
Production breaks on the path the agent never exercised

MCP gives the agent (and you) a receipt: sandbox timeline, step statuses, real provider-shaped IDs from the run — before opening the PR.

CI is the gate that keeps that proof from regressing on the next agent session.

What a workflow run actually checks

Not "does POST return 200 once."

Create a resource → mutate it → GET the same ID back
Trigger the webhook branch when state changes
Fail with a named step when auth or transitions break

That is integration testing — not HTTP cosplay.

What we stopped doing

Treating mocked unit tests as "integration covered"
Running only manual smoke before deploy
Letting agents merge Stripe/Resend/Clerk changes without a workflow receipt

Try it

Full write-up: API integration testing in CI

MCP + agent workflows: Agent API workflow testing

Install MCP in Cursor/Claude → run a Stripe workflow locally → paste the same npx fetchsandbox run step into Actions.

fetchsandbox.com · MCP setup

Our partners kept breaking on staging. So we gave them production access. (Don't do this.)

FetchSandbox — Thu, 25 Jun 2026 16:43:57 +0000

We whitelisted partners on production with read-only test accounts. That was attempt four — after docs-only onboarding, shared UAT, and IP-whitelisted staging all failed in different ways.

I work on an embed platform. Partners integrate our APIs for payments, identity verification, and onboarding flows. We burned six months on one question: how do partners test their integration before go-live?

Attempt 1: "Just use our docs"

We pointed partners at endpoints, auth, request/response examples. "You're good to go."

They'd hit something we didn't document, open a support ticket, wait two days, lose momentum. A few never came back. One went with a competitor because they "couldn't get a working test setup in a week."

The docs described what should happen. Partners had no way to see what actually happens until they wrote code, deployed, and hoped.

Attempt 2: Internal UAT box

Next: give partners our internal UAT environment. IP whitelisted, shared credentials. We told them "please don't break anything" without irony.

Worked for three weeks.

QA pushed a bad config on a Tuesday. The partner's integration broke. They spent two days debugging their own code before opening a ticket. By the time we figured it out, their engineering lead was cc'ing our VP.

Another team ran load tests on UAT the same week. Partner API calls timed out. Their CTO asked if our platform was "production-ready."

UAT was built for us to break things. We invited external partners into that mess.

Attempt 3: Staging with IP whitelisting

We carved out "partner staging" — same codebase, separate deployment, IP whitelisted per partner. Felt like the grown-up solution.

It wasn't.

IP whitelists became a full-time job. Every new partner meant firewall rules. Home IPs differed from office IPs. One CTO traveling couldn't hit the sandbox from his hotel. We debugged VPN configs at 11pm on a Thursday.

Test data went stale. Partners created orders for products that existed in production but not staging. "Your API returns 404 for product_id XYZ." Correct — nobody seeded staging in three weeks.

Deploys collided with demos. Engineers shipped to staging without checking if partners were testing. A deploy during a partner's live demo call gets brought up in quarterly business reviews.

Cost. Full production mirror for partner testing — database, compute, monitoring, on-call — used maybe 10 hours a week.

Attempt 4: Production access

I wish I was kidding.

Reasoning: staging is flaky, UAT is a disaster, whitelist them on production with read-only test accounts.

The API was stable. Partners were happy. For about a month.

One partner's integration bug created 400 orphaned records in production. Data team spent a weekend cleaning up.

Compliance asked why test payloads with fake PII hit the production database.

Maintenance meant coordinating downtime with partners "just running a few test calls."

We'd built the world's most expensive sandbox: production with IP whitelisting.

What we actually needed

Partners didn't need access to our infrastructure.

They needed an API that looked and acted like ours — same endpoints, schemas, auth patterns — but was completely separate. POST creates a resource. GET retrieves it. State transitions work. Nobody else's deploy breaks it.

Not a mock server — those are stateless; step 2 doesn't know about step 1. Not a shared staging box — that's everybody's problem. A dedicated sandbox generated from the same OpenAPI spec the real API uses.

This is half why I started working on FetchSandbox. Give it an OpenAPI spec, get a stateful sandbox — CRUD, state machines, webhook events, seed data. No infrastructure to maintain.

# Partner gets a sandbox from your spec
npx fetchsandbox generate ./your-api-openapi.yaml

curl https://your-api.fetchsandbox.com/v1/orders \
  -H "api-key: sandbox_abc123"
# → 200 OK, realistic seed data

fetchsandbox run your-api create-and-fulfill-order
# → ✓ Create order — 201
# → ✓ Add line items — 200
# → ✓ Submit for fulfillment — 200
# → ✓ Webhook: order.fulfilled fired

No staging to maintain. No IP whitelists. No production risk. Partners get their own URL, data, and credentials. Your team doesn't touch it.

When a partner asks "does this workflow work?" — they prove it themselves.

The time sink nobody tracks

If you run a partner API, count hours per month on:

Provisioning and maintaining test environments
Debugging "is your sandbox down?" tickets
Managing IP whitelists and VPN access
Re-seeding stale test data
Coordinating deploys around partner testing schedules

At my company: 20–30 hours across engineering and DevOps. For a problem that shouldn't exist.

Open the Stripe sandbox and run a workflow end-to-end — no signup: fetchsandbox.com/playground

Resend webhook reconciliation: map events to the right email record

FetchSandbox — Tue, 23 Jun 2026 20:02:20 +0000

This is a different failure mode from treating email.sent as delivered. That bug is here. This post is about what happens after you store the Resend email ID.

POST /emails returned 200. You saved the Resend email ID on your notification row. A minute later the webhook handler runs.

Then support asks: "Why does the dashboard say delivered when the address bounced?"

The send call worked. The reconciliation layer did not.

The bug is usually a flat status field

Most handlers start like this:

app.post("/webhooks/resend", async (req, res) => {
  const event = req.body;
  const emailId = event.data.email_id;

  const row = await db.notifications.findByResendId(emailId);
  if (!row) return res.sendStatus(200);

  if (event.type === "email.delivered") {
    await db.notifications.update(row.id, { status: "delivered" });
  }
  if (event.type === "email.opened") {
    await db.notifications.update(row.id, { status: "opened" });
  }
  if (event.type === "email.bounced") {
    await db.notifications.update(row.id, { status: "bounced" });
  }

  res.sendStatus(200);
});

Three separate problems hide in that shape:

Lookup failure — webhook arrives before your app finishes persisting the Resend ID, or the ID is stored on the wrong table. Handler returns 200, nothing updates, bug disappears until production.
Last-write-wins — email.opened overwrites delivered. Fine if you track engagement separately. Bad if status drives UX copy or retry logic.
No terminal states — email.bounced should win over delivered when events arrive out of order or retry. A single string field cannot express that without rules.

Separate delivery state from engagement

A pattern that survives production:

const TERMINAL = new Set(["bounced", "complained", "failed"]);

function nextDeliveryState(current, incoming) {
  if (TERMINAL.has(current)) return current;
  if (incoming === "bounced" || incoming === "complained") return incoming;
  if (incoming === "delivered") return current === "sent" ? "delivered" : current;
  if (incoming === "sent") return current ?? "sent";
  return current;
}

app.post("/webhooks/resend", async (req, res) => {
  const { type, data } = req.body;
  const emailId = data.email_id;

  const row = await db.notifications.findByResendId(emailId);
  if (!row) {
    await db.webhookInbox.insert({ provider: "resend", emailId, type, payload: req.body });
    return res.sendStatus(200);
  }

  if (type === "email.opened" || type === "email.clicked") {
    await db.emailEvents.insert({ notificationId: row.id, type });
    return res.sendStatus(200);
  }

  const next = nextDeliveryState(row.deliveryStatus, type.replace("email.", ""));
  await db.notifications.update(row.id, {
    deliveryStatus: next,
    lastEventType: type,
    lastEventAt: new Date(),
  });

  res.sendStatus(200);
});

Notes that matter in real apps:

email.opened is not delivery. Log it separately unless your product truly treats opens as delivery confirmation.
Store unmatched webhooks. If the event arrives before the send transaction commits, you need a replay path — not a silent 200.
Idempotency key = Resend email ID + event type. Retries are normal.

The test most teams skip

Unit tests mock one webhook payload. That proves parsing, not reconciliation.

The integration test that catches the bug:

create internal notification row
POST /emails and persist Resend email ID on that row
deliver email.sent, then email.delivered
deliver email.opened and assert delivery status did not regress
deliver email.bounced on a second send and assert terminal state wins

Step 4 and 5 are where flat status fields break.

Receipts

Real Resend send workflow run against FetchSandbox, captured 2026-06-07:

POST /emails                                       → 200
  id: 4d9acb24-5913-4e02-8b1a-a18ed10a6fad

GET  /emails/4d9acb24-5913-4e02-8b1a-a18ed10a6fad  → 200

webhook events verified:
  email.sent
  email.delivered

Full timeline: fetchsandbox.com/runs/8e4fe8e9f9?flow=run_4062510e-268e-451f-b5f4-e8844eef42d5

That run proves the provider lifecycle through email.delivered. Your app still needs its own test for mapping those events back to the correct internal row and for terminal bounce/complaint handling.

Runnable preflight (optional)

Before wiring production webhooks, I run the send workflow from Cursor with FetchSandbox MCP:

validate resend email workflow with fetchsandbox

Workflow page: fetchsandbox.com/docs/resend

Resend sandbox page (mock API + webhook replay): fetchsandbox.com/resend

That does not replace Resend's real domain, API key, or production webhook endpoint. It just makes the lifecycle obvious before you commit your local state machine.

Test AgentMail multi-tenant webhooks before they leak across tenants

FetchSandbox — Tue, 16 Jun 2026 17:16:05 +0000

The dangerous AgentMail webhook bug starts with a response that looks completely fine.

POST /v0/webhooks -> 200 OK

The response body contains the webhook object. It includes your URL, event types, and usually enough fields to make the integration feel done.

But in a multi-tenant app, the field that matters is inbox_ids. If that array is silently dropped, your webhook is no longer scoped to the tenant inbox you meant to subscribe. It is subscribed broadly, and every tenant's message events can start flooding the same endpoint.

That is not a noisy failure. It is worse. The subscribe call worked. Your endpoint receives events. The logs look active. The leak is in the scope.

If your handler routes by webhook ID before it checks the inbox ID, the wrong customer can look like a valid event. That is the kind of bug that passes observability and fails privacy.

Why unit tests miss it

Most unit tests stub the webhook create call by echoing back exactly what the app sent.

request.inbox_ids -> response.inbox_ids

That proves your client serialized the payload. It does not prove AgentMail persisted the scope.

The test passes. CI passes. The code ships. Then production receives events for inboxes that do not belong to the tenant that configured the webhook.

The annoying part is that the local test is not obviously wrong. It has the right endpoint, the right payload shape, and the right assertion against the create response. It just trusts the wrong copy of the object.

The pattern is reconcile-after-write

The fix is small enough to name:

POST -> GET -> diff

Treat the POST response as untrusted. It may be your request echoed back. Treat the later GET as the provider state. That is what AgentMail actually stored.

The diff is the bug.

const sentInboxIds = ["inbox_tenant_123"];
const created = await agentmail.post("/v0/webhooks", {
  url: "https://app.example.com/webhooks/agentmail",
  event_types: ["message.delivered", "message.bounced"],
  inbox_ids: sentInboxIds,
});
const stored = await agentmail.get(`/v0/webhooks/${created.id}`);
const got = [...(stored.inbox_ids ?? [])].sort();
const want = [...sentInboxIds].sort();
if (JSON.stringify(got) !== JSON.stringify(want)) {
  throw new Error("AgentMail webhook scope drifted");
}

That is the whole check. It is not a bigger mock. It is a read after the write.

Why the agent caught it

The reason a coding agent caught this where the unit test did not is not that it was smarter about webhooks. It ran a better-shaped test.

AgentMail has a curated webhook_lifecycle_create_read_delete workflow in FetchSandbox. When the agent ran it through the FetchSandbox MCP server, the workflow forced the boring step people skip:

create inbox
subscribe webhook scoped to that inbox
read webhook back
compare inbox_ids
delete webhook

The workflow shape matches the bug shape. A stubbed unit test checks what your app meant to send. The workflow checks what the provider kept.

This is not only AgentMail

The same bug appears anywhere a control-plane API returns an object that looks like the request you sent:

IAM policies
ACL rules
draft resources
webhook subscriptions
tenant-scoped notification settings

If a create call returns 200, but the security or tenancy boundary matters, do not stop at the create response.

Read it back. Diff the fields that protect the boundary. Fail the test before the provider starts sending another tenant's events to your endpoint.

Run the brownfield demo

The brownfield-agentmail-demo repo shows this end to end if you want to run it against the workflow yourself.

The important part is not AgentMail-specific. The habit is: after a write that configures scope, reconcile what the provider persisted.

A webhook that passes create tests can still be the webhook that leaks across tenants.

FetchSandbox's AgentMail workflow docs include the create, read, and teardown path.

Clerk webhook replay can create duplicate users unless your sync is idempotent

FetchSandbox — Mon, 15 Jun 2026 13:57:02 +0000

The login worked. The webhook worked. Then the same event arrived again.

Clerk webhooks are easy to treat like a notification:

user.created arrived
create a local user
done

That works until the event is retried, replayed, or delivered after another part of your app already created the user.

Then your app has two local rows for the same Clerk user. Or one row has the right email and the other has the right metadata. Or your onboarding job runs twice and sends the same welcome email twice.

The bug is not "Clerk webhooks are unreliable." Retrying delivery is normal webhook behavior.

The bug is trusting delivery count instead of provider state.

The narrow fix

Your webhook handler should not mean:

on user.created -> insert user

It should mean:

on user.created -> upsert by clerk_user_id

The important field is the stable provider ID, not the local row ID and not the email address.

await db.user.upsert({
  where: { clerkUserId: event.data.id },
  update: {
    email: event.data.email_addresses?.[0]?.email_address,
    firstName: event.data.first_name,
    lastName: event.data.last_name,
    clerkUpdatedAt: new Date(event.data.updated_at),
  },
  create: {
    clerkUserId: event.data.id,
    email: event.data.email_addresses?.[0]?.email_address,
    firstName: event.data.first_name,
    lastName: event.data.last_name,
    clerkUpdatedAt: new Date(event.data.updated_at),
  },
});

That one constraint does most of the work:

UNIQUE(clerk_user_id)

Now a replay updates the same user instead of creating a second one.

The part people skip

The handler can still be wrong even after the upsert.

If your app grants access, creates a workspace, starts a trial, or sends email inside the same handler, those side effects need their own idempotency rule too.

For example:

workspace owner = clerk_user_id
welcome email key = clerk_event_id
trial grant key = clerk_user_id + plan_id

The user row is only one piece of local state.

How I would test it

I would test the same user.created event twice.

First delivery:

local user count: 0 -> 1
workspace count: 0 -> 1

Replay:

local user count: 1 -> 1
workspace count: 1 -> 1

That is the whole test.

FetchSandbox has a Clerk sandbox and runnable Clerk workflow docs for this exact kind of replay check. It is also where a stateful sandbox is more useful than a static mock response: the second delivery is the thing you need to observe.

Takeaway

Do not write Clerk webhook handlers as if events happen once.

Write them as reconciliation:

given this Clerk user ID,
what should my app state be now?

If the answer changes when the same event arrives twice, the bug is already there. Production is just waiting to replay it.

WorkOS directory sync webhooks need reconciliation after Okta group renames

FetchSandbox — Thu, 11 Jun 2026 14:25:12 +0000

The bug shows up two minutes after the customer says, "we only renamed the group."

In Okta, the admin changed Enterprise Admins to Platform Admins. WorkOS sent dsync.group.updated. Your webhook handler saw the new name. The cache updated. Everything looked normal.

Then support gets the screenshot: half the enterprise customer's users can access the gated feature. Half can't. The senior employees are the first ones broken because they sit in the group your app treats as the entitlement source.

The webhook is not the final state

The tempting handler is simple:

type WorkOSGroupUpdated = {
  event: "dsync.group.updated";
  data: {
    id: string;
    name: string;
  };
};

export async function handleGroupUpdated(event: WorkOSGroupUpdated) {
  await groupsCache.set(event.data.id, {
    name: event.data.name,
    updatedAt: new Date(),
  });
}

That works only if the webhook payload is the same thing as the directory state your feature gate should trust.

It is not.

Okta sends SCIM PATCH operations to WorkOS in batches that do not necessarily match the order an admin sees in Okta's UI. WorkOS directory sync webhooks fire as individual events. They tell you something changed. They do not give you a complete, ordered snapshot of the group and every user's current membership.

If your app updates downstream feature gating from the webhook payload, it can partially reconcile. One cache entry has the new group name. Another user's group membership still reflects the old world. The feature gate now depends on which stale value a request happened to read.

Treat dsync webhooks as invalidation

For feature gating, a dsync.group.updated or dsync.user.updated webhook should invalidate local state. It should not be the state.

The safer handler does two reads after every relevant webhook:

type DSyncEvent = {
  event: "dsync.group.updated" | "dsync.user.updated";
  data: {
    id: string;
  };
};

export async function handleDirectorySyncEvent(event: DSyncEvent) {
  if (event.event === "dsync.group.updated") {
    const group = await workos.directorySync.getDirectoryGroup(event.data.id);
    const users = await workos.directorySync.listDirectoryUsers({
      directory: group.directoryId,
    });

    await featureGateStore.reconcileDirectoryGroup({
      groupId: group.id,
      groupName: group.name,
      users: users.data,
    });
  }

  if (event.event === "dsync.user.updated") {
    const users = await workos.directorySync.listDirectoryUsers();

    await featureGateStore.reconcileDirectoryUsers(users.data);
  }
}

The important part is not the SDK shape. It is the rule: ignore the webhook payload's name and groups for downstream feature gating. Use the payload ID to know what to re-fetch. Then read from GET /directory_groups/{id} and GET /directory_users.

Those reads are the source of truth your cache reconciles against.

What to test

The test is not "did we receive dsync.group.updated?"

The test is:

Start with a group that controls a feature gate.
Rename the group in the identity provider.
Receive the group update webhook.
Re-fetch the group by ID.
Re-fetch directory users.
Rebuild the local entitlement view from the reads, not the webhook payload.

That is the path that catches the bug. A unit test around the webhook JSON does not.

Receipts

Here's a real WorkOS directory group reconciliation workflow run against the FetchSandbox sandbox, captured 2026-06-11:

GET /directories → 200
GET /directory_groups → 200
GET /directory_groups/54483a43-5333-489d-8930-2a549bce4de9 → 200
GET /directory_users → 200

Sandbox ID: a8d236f155

Flow run ID: run_227a570b-1c7d-42d8-ad8d-af1c09c67b60

Full timeline: fetchsandbox.com/runs/a8d236f155?flow=run_227a570b-1c7d-42d8-ad8d-af1c09c67b60

The group ID 54483a43-5333-489d-8930-2a549bce4de9 is the value the webhook should use to trigger reconciliation. The follow-up GET /directory_users is what keeps feature gating from depending on a partial webhook payload.

How to test this without waiting on Okta

For WorkOS directory sync, the finish line is not "webhook received." It is "the app rebuilt its local entitlement view from the directory state."

We use FetchSandbox to run the directory group reconciliation workflow before wiring the production handler. The point is not to replace Okta or WorkOS testing. The point is to make the cache rule obvious before a real enterprise customer renames the group your feature gate depends on.

Clerk JWT 401 on the server? Check these 3 env vars first

FetchSandbox — Tue, 09 Jun 2026 14:58:41 +0000

A Clerk login can look fine in the browser and still fail on your server with a 401.

The frontend has a user. getToken() returns something. The request includes:

Authorization: Bearer <token>

Then FastAPI, Express, or your API route rejects it.

Before rewriting your auth code, check this:

CLERK_PUBLISHABLE_KEY
CLERK_SECRET_KEY
CLERK_JWT_ISSUER

They need to come from the same Clerk instance.

The bug

This can happen when you have multiple Clerk projects open:

Frontend:
  CLERK_PUBLISHABLE_KEY = pk_test_from_project_A

Backend:
  CLERK_SECRET_KEY = sk_test_from_project_B
  CLERK_JWT_ISSUER = https://project-c.clerk.accounts.dev

Each value can look valid by itself.

But the token was minted by one Clerk instance, and your backend is trying to verify it against another one. That is enough for server-side verification to fail.

You might see:

JWT issuer mismatch
invalid issuer
JWKS key not found
invalid signature
401 unauthorized

Different library, same root issue.

The quick fix

Open one Clerk dashboard project and copy all auth values again from the same place.

Then restart both servers.

# frontend
NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY=pk_test_...

# backend
CLERK_SECRET_KEY=sk_test_...
CLERK_JWT_ISSUER=https://same-instance.clerk.accounts.dev

Do not only refresh the browser. Dev servers often keep old env values until the process restarts.

Why agents miss this

An AI coding agent can write the shape of a Clerk integration:

add middleware
read bearer token
verify JWT
sync user
protect route

But it will not know you pasted keys from two Clerk projects unless it checks the running path.

This is why I like running a small auth workflow before wiring the app.

Receipt

I ran FetchSandbox's Clerk user_signup workflow as the receipt layer:

Workflow: clerk/user_signup
Result: passed
Steps: 4/4
Duration: 77.44 ms
Verified webhooks: user.created, user.updated
Timeline: https://fetchsandbox.com/runs/7aa06cff84?flow=run_b8399f77-352c-4fe4-ad50-24e3060f8d8d

That workflow proves the user lifecycle surface:

POST /users
GET /users/{id}
PATCH /users/{id}
verify user.created + user.updated

It does not replace your real Clerk project. It gives your agent or teammate a runnable auth flow before they touch app code.

Takeaway

If Clerk works in the browser but your server returns 401, check the boring thing first:

publishable key
secret key
issuer URL

Same instance. Same environment. Then restart.

FetchSandbox MCP setup if you want the agent to run the workflow first:

{
  "mcpServers": {
    "fetchsandbox": {
      "command": "npx",
      "args": ["-y", "fetchsandbox-mcp"]
    }
  }
}

Test Clerk auth lifecycle flows without production users

FetchSandbox — Tue, 02 Jun 2026 04:22:48 +0000

Most auth bugs do not come from the login button.

They show up after login:

a user is created in Clerk but not in your app database
a session expires while your UI still thinks the user is active
a webhook arrives late or twice
a deleted user still has local app state
JWT verification works locally, then fails once keys rotate

That is the part I want API integration agents to test before they write app code.

I have been working on FetchSandbox MCP support for Clerk-style auth workflows so an IDE agent can run the auth lifecycle in a sandbox first, then wire the app around the behavior it just proved.

The narrow workflow

For a Clerk integration, the useful test is not just:

Can I create a user?

The useful test is closer to:

Can my app survive the auth lifecycle?

That includes:

Create or receive a user
Create a session
Verify the token/session state
Receive a webhook event
Reconcile that event into app state
End or revoke the session
Confirm the app no longer treats the user as active

In real apps, each of those steps tends to live in a different place: middleware, webhook handlers, user tables, session checks, background jobs, and sometimes frontend state.

That is exactly where AI coding agents need more than docs.

Where normal AI coding breaks down

If you ask an agent to "add Clerk auth," it can usually generate plausible code:

Add middleware.
Read userId.
Create a webhook route.
Verify signatures.
Sync the user.

That is useful, but it is not proof.

The agent still has to guess:

which event shape matters
what your app should store
what should happen when the same event arrives again
what happens when the session ends
whether user deletion should cascade, archive, or soft-disable local data

Those are integration questions, not syntax questions.

What FetchSandbox MCP changes

FetchSandbox gives an IDE agent a runnable API workflow environment.

Instead of only reading docs, the agent can:

route the user's intent to a Clerk-style workflow
execute the auth lifecycle in a sandbox
inspect request and response payloads
inspect webhook event shapes
identify which app files need to handle each transition
summarize proof and next steps before editing code

The ideal loop looks like this:

developer intent
↓
FetchSandbox MCP route
↓
run auth workflow
↓
inspect payloads and state transitions
↓
wire app code
↓
summarize proof + remaining edge cases

That makes the agent less likely to invent a happy-path-only integration.

Example prompt

From Cursor, Claude Code, or another MCP-capable IDE:

Help me integrate Clerk auth into this app.
Before editing code, use FetchSandbox to prove the user/session lifecycle and webhook payload shape.

The important part is "before editing code."

For auth, that order matters.

The failure modes worth testing

For a Clerk-style integration, I would rather see an agent test these before it changes the app:

User created

Does the app create or update the local user record correctly?

Session created

Does the app treat the session as active only when the provider state says it is active?

Session ended

Does the app stop trusting stale session state?

User deleted

Does the app remove access without accidentally deleting data that should be retained?

Webhook replay

If the same webhook arrives twice, does the app stay idempotent?

Signature verification

Does the webhook handler verify the signed payload before trusting the event?

None of these are solved by a code snippet alone.

They need a workflow.

Why this matters for AI agents

AI coding agents are very good at producing integration code.

But API integrations need an execution layer.

Without that layer, the agent is forced to infer behavior from docs and examples. With FetchSandbox MCP, the agent can run a workflow first and then use the result as context.

The difference is subtle but important:

"Here is auth code that should work."

versus:

"Here is the auth lifecycle I ran, the payloads I saw, and the code paths I wired."

That is the workflow I want for API integrations.

Setup

Add FetchSandbox MCP to your IDE:

{
  "mcpServers": {
    "fetchsandbox": {
      "command": "npx",
      "args": ["-y", "fetchsandbox-mcp"]
    }
  }
}

Then ask the agent to prove the provider workflow before implementation.

Use FetchSandbox MCP to test the Clerk auth lifecycle before wiring the app.

FetchSandbox:

https://fetchsandbox.com

MCP:

https://fetchsandbox.com/mcp

Takeaway

Auth integrations do not end at login.

The lifecycle is the contract:

users
sessions
tokens
webhooks
app-state reconciliation

If an AI agent is going to wire auth into your app, it should be able to run that lifecycle first.

Connect the provider.

Prove the workflow.

Ship the integration.

DEV Community: FetchSandbox

Why Paddle's subscription.activated arrives before subscription.created

What actually happened

Why this happens

The fix

Testing this is the real problem

The takeaway

Webhook handlers that silently return 402: the event ordering bug most test suites miss

We planted a Descope privilege-escalation bug — can your coding agent catch it?

The 20-minute test drive

What a real catch looks like

The other planted bugs

The part where you tell us it fell flat

How to test Descope auth without real keys

Our CI mocked Stripe. Production still broke on step two.

What unit tests miss

What we run now — MCP locally, same workflow in CI

Why MCP belongs in the loop

What a workflow run actually checks

What we stopped doing

Try it

Our partners kept breaking on staging. So we gave them production access. (Don't do this.)

Attempt 1: "Just use our docs"

Attempt 2: Internal UAT box

Attempt 3: Staging with IP whitelisting

Attempt 4: Production access

What we actually needed

The time sink nobody tracks

Resend webhook reconciliation: map events to the right email record

The bug is usually a flat status field

Separate delivery state from engagement

The test most teams skip

Receipts

Runnable preflight (optional)

What to read next

Test AgentMail multi-tenant webhooks before they leak across tenants

Why unit tests miss it

The pattern is reconcile-after-write

Why the agent caught it

This is not only AgentMail

Run the brownfield demo

Clerk webhook replay can create duplicate users unless your sync is idempotent

The narrow fix

The part people skip

How I would test it

Takeaway

WorkOS directory sync webhooks need reconciliation after Okta group renames

The webhook is not the final state

Treat dsync webhooks as invalidation

What to test

Receipts

How to test this without waiting on Okta

Clerk JWT 401 on the server? Check these 3 env vars first

The bug

The quick fix

Why agents miss this

Receipt

Takeaway

Test Clerk auth lifecycle flows without production users

The narrow workflow

Where normal AI coding breaks down

What FetchSandbox MCP changes

Example prompt

The failure modes worth testing

User created

Session created

Session ended

User deleted

Webhook replay

Signature verification

Why this matters for AI agents

Setup

Takeaway