DEV Community: FetchSandbox

undefined === undefined — the auth bypass your AI wrote into your checkout

FetchSandbox — Thu, 13 Aug 2026 17:40:08 +0000

TL;DR

An auth check that uses === where both sides can be undefined is fail-open — a missing config turns it from locked to open.

The one secret missing from .env.example was the internal auth token. Unset in a deploy, undefined === undefined let every anonymous request through.

Default to deny: validate types, reject empty, compare in constant time, and write the test that sends no credentials with the env var unset.

The coding agent handed me an auth middleware. It passed review. I almost shipped it.

function protect(req, res, next) {
  if (req.headers['x-internal-token'] === process.env.INTERNAL_API_TOKEN) return next();
  return requireAuth()(req, res, next);
}

Four lines, and it reads fine. An internal service calls with a shared token and skips the login flow; everyone else falls through to real auth. Then I checked what happens when the token isn't set.

What the check actually did

INTERNAL_API_TOKEN was the one secret missing from .env.example. Every other key was there — Stripe, Clerk, Paddle — but not this one. So on a deploy where nobody thought to set it, process.env.INTERNAL_API_TOKEN is undefined.

Now a normal browser request comes in. It doesn't send an x-internal-token header, so req.headers['x-internal-token'] is also undefined.

undefined === undefined → true → return next().

Auth bypassed. Every anonymous request to /checkout is treated as a trusted internal call. The endpoint is safest when the token is set and wide open when it's missing — which is exactly backwards. A misconfigured auth check should get more restrictive, not less.

And there's a quieter problem even when the token is set: === on a secret isn't constant-time. It short-circuits on the first mismatched byte, so response timing leaks the token one character at a time.

The real check

Fail closed:

const crypto = require('crypto');

function hasInternalToken(req) {
  const expected = process.env.INTERNAL_API_TOKEN;
  if (typeof expected !== 'string' || expected.length === 0) return false; // no token configured → deny
  const provided = req.headers['x-internal-token'];
  if (typeof provided !== 'string' || provided.length === 0) return false;
  const a = Buffer.from(provided), b = Buffer.from(expected);
  if (a.length !== b.length) return false;
  return crypto.timingSafeEqual(a, b);
}

function protect(req, res, next) {
  if (hasInternalToken(req)) return next();
  return requireAuth()(req, res, next);
}

No configured token means no bypass. An empty or missing header means no bypass. The compare is constant-time, and length is checked first (timingSafeEqual throws on length mismatch). Then add INTERNAL_API_TOKEN to .env.example so it can never be silently omitted again.

Why agents write it this way

The happy path looks correct. Token set, caller sends the right one, x === y is true — review sees a plausible auth check and moves on. The model optimizes for "works when configured," and === fails open by default. It never had to reason about the unset case, because the unset case isn't in the example the code was written against.

Humans do this too — the fail-open default, the === on a secret, the config that's "obviously" always set. Agents just do it faster and ship it greener.

Fail closed, and prove it

The tell isn't in the code. It's in the test you didn't write. Send a request with no credentials and the env var unset, and assert it's rejected:

delete process.env.INTERNAL_API_TOKEN;
const res = await request(app).post('/checkout').send({ seats: 5 });
expect(res.status).not.toBe(200); // must be 401/302 — not "welcome in"

If an unauthenticated request with no token configured gets a 200, your auth is open — and no amount of green on the happy path will tell you. The bug lives entirely in the case the tests never exercise.

What I changed in the habit

Auth defaults to deny. Every branch that grants access starts from "no."
Validate both sides — type is string, length > 0. undefined === undefined is not authentication.
Constant-time compare for any secret (crypto.timingSafeEqual), never ===.
Write the misconfigured-env test — unset the token, send nothing, assert rejected.
Every secret the code reads goes in .env.example. A missing one isn't a config gap; here it was the whole lock.

I found this the way I find most of them: reproduced it on the real code — an anonymous request returned 200 — applied the fix, and confirmed it flipped to a 302, with FetchSandbox. The bug was never in the happy path. It was in the case the code never had to handle.

Which is the rule every auth check should start from: fail closed.

Green passed. The fix granted zero seats.

FetchSandbox — Tue, 11 Aug 2026 14:05:17 +0000

TL;DR

"Did the bug stop?" is a weak invariant. An over-suppressing fix — grant nothing — also stops the bug.

Assert the exact correct end state: after N identical deliveries, seats == the purchased amount. Not more. Not zero.

A verifier that cannot reject a deliberately broken fix is not a verifier. It is a green light.

The coding agent handed me a patch. CI went green. I almost merged it.

The original bug was ugly and familiar. A Stripe webhook handler granted seats on invoice.paid (or whatever your equivalent is — checkout.session.completed, customer.subscription.updated, pick your poison). It was not idempotent. Stripe retried. The handler granted again.

A five-seat purchase became 5, then 10, then 15. Same event. Same customer. Three deliveries.

That is the bug everyone warns you about. At-least-once delivery. Dedupe on event id. INSERT … ON CONFLICT DO NOTHING. You have heard the sermon.

So I asked the agent to fix it. It wrote a patch. My verifier ran the retry series and graded the result green.

Then I looked at the seats.

delivery     buggy handler     agent "fix"      what we wanted
1            5                 0                5
2            10                0                5
3            15                0                5

The count had stopped growing. True. It had also stopped granting.

Customers who paid got nothing. Including on the first, legitimate delivery. The bug was "fixed" the way you fix a leaky pipe by shutting off the water main.

What the check actually asserted

The verifier was not stupid in a cartoon way. It did replay the webhook. It did look at a side effect. It did not just grep the diff for idempotent.

It asserted the thing I had complained about: the count should not keep climbing.

Something like this:

// Weak: "the bug stopped"
const series = seatsAfterEachDelivery(handler); // e.g. [0, 0, 0]

expect(series[2]).toBeLessThanOrEqual(series[0]);
expect(new Set(series).size).toBe(1); // "stable across retries"

[5, 10, 15] fails. Good.

[5, 5, 5] passes. Also good.

[0, 0, 0] passes. That is the hole.

Any monotonicity check, any "didn't grow," any "delta is zero after the first call" that never pins the value will accept the over-suppressing patch. Delete the grant. Swallow the event. Return 200 and write nothing. The graph is flat. Green.

If you have ever written expect(errors).toHaveLength(0) and then watched someone delete the code that could error, you have met this family.

The real invariant

Idempotency is not "nothing happens twice." It is "the same request leaves the system in the same correct state."

For this handler, after N identical deliveries of a five-seat purchase, seats must equal exactly five. Not fifteen. Not zero.

// Exact: after N identical deliveries, seats == purchased amount
const series = seatsAfterEachDelivery(handler);

expect(series).toEqual([5, 5, 5]);

That one line rejects [5, 10, 15] and [0, 0, 0]. It accepts [5, 5, 5].

When I switched the assert, the zero-grant patch went red. The agent tried again. The patch that actually deduped — grant once, ignore the retries — produced [5, 5, 5] and stayed green.

Same deliveries. Same metric. Different question.

"Did it stop doing the bad thing?" vs "Is the world in the state a correct implementation would leave it in?"

Why agents fail this way

I do not think the model was being clever. I think it was being literal.

You said: stop double-granting. The shortest path to "the number does not increase" is "the number never moves." Skip the write. Make the insert always look like a conflict. Short-circuit before grantSeats. Plenty of ways to get a flat series.

Humans do this too under time pressure. Feature flags that default off. if (false) around the dangerous block. Tests that mock the collaborator into a no-op. Agents just do it faster, and they will happily stop at the first green.

Your CI is an optimization target. If the loss function is "bug symptom gone," over-suppression is a local minimum. If the loss function is "exact end state," that minimum disappears.

This is not a Stripe trivia item. It is any side-effecting handler you let an agent touch:

A refund webhook that "fixes" double-refunds by refunding $0.
A provisioner that "fixes" duplicate users by creating none.
A retry queue that "fixes" duplicate emails by sending nothing, including the first time.

The shape is always the same. Symptom: too much. Naive fix: zero. Correct fix: once.

Positive control, negative control

Here is the part I now require before I trust a verifier — mine, yours, an agent's, a CI job.

Positive control. A known-good implementation must pass. For me that was the idempotent grant: [5, 5, 5]. If your exact-count assert fails on a patch you already believe is right, the assert is wrong, not the patch.

Negative control. A known-bad implementation must fail. I keep a deliberately broken patch around: the zero-grant one, or the original double-grant, or both.

expect(seatsAfterEachDelivery(buggy)).toEqual([5, 5, 5]);
// fail: [5, 10, 15]

expect(seatsAfterEachDelivery(zeroGrant)).toEqual([5, 5, 5]);
// fail: [0, 0, 0]   ← this is the control most suites skip

expect(seatsAfterEachDelivery(idempotent)).toEqual([5, 5, 5]);
// pass

If the zero-grant still greens, you do not have a proof. You have a check that the symptom you first noticed got quieter.

I used to stop at the positive control. "The good fix passes, ship it." That is how [0, 0, 0] got a green. The suite had never been shown a lying patch.

This is the same idea as a mutation test, just less academic. You do not need a framework. You need one broken cousin of the fix, and a gate that refuses to call itself a verifier until that cousin fails.

What I changed in the habit

I still replay the webhook three times. That part was never the mistake. The mistake was scoring the replay with a bound instead of an equality.

The checklist I use now, when an agent "fixes" a production-shaped bug:

Write the end state in English first. "After three identical invoice.paid deliveries for a 5-seat price, seat_limit == 5."
Put that number in the assert. Not <=. Not "unchanged after first." The number.
Run the original bug. It must fail that assert.
Run a spiteful fix (no-op, always-conflict, grant-zero). It must fail too.
Then run the agent's patch.

Step 4 is the one I had skipped. It is also the cheapest. You can write the no-op in thirty seconds. If your suite cannot fail it, do not let the suite pass the agent.

I hit this while building FetchSandbox, replaying Stripe deliveries against persistent seat state instead of a one-shot fixture.

The transferable rule is not about Stripe, and it is not about my tooling. When you verify an AI-written fix, "did the bug stop?" is the wrong question. Ask whether the system is in the exact state a correct implementation would leave it in — and prove your question can still say no.

You can't code-review a webhook retry

FetchSandbox — Tue, 04 Aug 2026 17:51:20 +0000

My coding agent wrote a Stripe webhook handler a few weeks ago. Signature check, event type check, fulfillment call, clean 200. The diff read like something I'd write on a good day. I approved it in about ninety seconds.

app.post("/webhooks/stripe", async (req, res) => {
  const event = verifySignature(req);
  if (event.type === "charge.succeeded") {
    await creditCustomer(event.data.object);
  }
  res.sendStatus(200);
});

Every line is correct. And the first time Stripe delivers that event twice — which it's allowed to do, and eventually will — the handler credits the customer twice.

Reading can't catch it

The bug isn't on any line, so no amount of reading finds it. Each statement is fine on its own. What's wrong is an assumption living between the lines: that every event arrives exactly once. Stripe promises at-least-once delivery. Retries and duplicates are documented, normal behavior.

Code review checks the diff against the traffic you can imagine. I imagined one clean charge.succeeded. So did the agent — the quickstart code it learned from imagined one too.

CI passed for the same reason

My tests replayed the events I thought of. The fixture file had exactly one copy of each event, because I wrote the fixture file. Nobody writes the test where the same event lands twice nine minutes apart. The suite was my assumptions checking my assumptions.

I spent three years on the developer platform of a very large payments company. This exact bug got past good reviewers more times than I can count. It was never a talent problem. Reading is the wrong tool for this class of bug.

The review that actually works

What caught it was running the failure. Deliver the event, deliver it again, watch the customer get credited twice. Add the dedupe on the event ID, run it again, watch it hold. Ten minutes, and the review went from "looks right" to "proved right."

That loop — reproduce the failure, fix it, rerun, keep the receipt — is what I built FetchSandbox to run from your IDE over MCP. The sandbox fires the real charge lifecycle, including the duplicate delivery your fixture file doesn't have, so your agent proves the handler before your customers do.

If you'd rather judge it on real code than my word, there's a webhook-dedupe bug planted in a Stripe app in our open playground, waiting to be caught: github.com/fetchsandbox/playground.

Why Paddle's subscription.activated arrives before subscription.created

FetchSandbox — Thu, 30 Jul 2026 13:58:08 +0000

You'd think events fire in the order things happen. They don't.

I was building a Paddle integration last week. Subscription billing, nothing fancy. Customer clicks buy, Paddle handles checkout, my app gets webhooks and updates the database.

The flow should be simple:

Customer completes checkout
subscription.created fires
subscription.activated fires
My app inserts a row on .created, updates status on .activated

That's what I built. It worked great in my head.

What actually happened

In production, subscription.activated arrived before subscription.created about 30% of the time.

My handler did a database insert on subscription.created:

case 'subscription.created':
  await db.insert(subscriptions).values({
    paddleId: event.data.id,
    status: 'created',
    customerId: event.data.customer_id,
  });
  break;

And an update on subscription.activated:

case 'subscription.activated':
  await db.update(subscriptions)
    .set({ status: 'active' })
    .where(eq(subscriptions.paddleId, event.data.id));
  break;

When .activated arrived first, the update found zero rows. No error, no exception. The WHERE clause just matched nothing. The update silently did nothing.

Then .created arrived and inserted the row with status created. But the .activated event was already gone. So the subscription was stuck in created status forever.

Customers had paid. Paddle showed them as active. My app showed them as pending. Support tickets started coming in.

Why this happens

Paddle does not guarantee webhook delivery order. Their docs mention it briefly but it's easy to miss when you're focused on the API endpoints.

The events are fired from different internal services. subscription.created comes from the subscription service. subscription.activated comes from the billing service after payment confirmation. They are async. They race.

This is not unique to Paddle either. Stripe has the same problem with payment_intent.created vs charge.succeeded. Most payment providers have some version of this.

The fix

The handler needs to be idempotent and order-independent. Every event should be able to create or update:

case 'subscription.created':
case 'subscription.activated':
  const status = event.event_type === 'subscription.activated' 
    ? 'active' : 'created';

  await db.insert(subscriptions)
    .values({
      paddleId: event.data.id,
      status,
      customerId: event.data.customer_id,
    })
    .onConflictDoUpdate({
      target: subscriptions.paddleId,
      set: { 
        status: sql`CASE WHEN ${status} = 'active' THEN 'active' ELSE ${subscriptions.status} END`
      },
    });
  break;

The key parts:

Both events can create the row if it doesn't exist
On conflict, active always wins over created regardless of arrival order
No silent failures, no missing updates

Testing this is the real problem

The ordering bug is easy to fix once you know about it. The hard part is reproducing it during development.

You can't control the order Paddle sends webhooks. You can't make .activated arrive first on demand. In testing you might run through the flow 20 times and the events always arrive in order. Then in production with real network latency and load, they don't.

I ended up testing this by sending the webhook events manually in the wrong order against a local sandbox. activated first, then created. Immediately saw the bug. Fixed it in 10 minutes.

The debugging in production took 4 hours.

If you're integrating Paddle or any payment provider with webhooks, test with events arriving in every possible order. Not just the happy path order from the docs.

Test Paddle webhook ordering in a sandbox →

The takeaway

Webhook events are not a queue. They are concurrent messages from different services that happen to be about the same thing. Your handler has to treat every event as potentially the first one it sees for that resource.

If your handler has an insert for one event type and an update for another, you have this bug. You just haven't hit it in production yet.

Webhook handlers that silently return 402: the event ordering bug most test suites miss

FetchSandbox — Tue, 21 Jul 2026 17:40:10 +0000

How to prove a handler is order-safe using permutation fuzzing

You're testing your Paddle webhook handler. You've got a test for subscription.created and subscription.activated, and maybe another one where you send them out of order. Looks good. All green. But weeks later, a customer hits a 402 on subscription.activated. No error in your logs. No alert. Just a failed charge and a confused user.

The problem isn't your test suite, it's the order of events.

A Paddle webhook handler that does an INSERT on subscription.created and an UPDATE on subscription.activated will silently return 402 when subscription.activated arrives first. The row doesn't exist yet. No error is raised. But if you send them in the "happy path" order, it works. Your test suite never sees the failure because it only runs one sequence. The bug is invisible until it hits production.

To prove a handler is order-safe, we built a permutation fuzzer in fetchsandbox MCP. It fires the same events in every permutation, including duplicates, and checks if the final state is the same in all of them. If it is, the handler is order-safe. If not, it returns the exact sequence that breaks it.

Here's what it found for the Paddle handler above:

When you send subscription.activated before subscription.created: the handler returns a 402.
When you send them in the "happy path" order: it returns a 200.
The fuzzer returns order_independent=False and shows you the diverging sequence.

After fixing the handler to use an upsert instead of separate INSERT and UPDATE, the fuzzer returns order_independent=True across every permutation and duplicate.

If you test API integrations in sandboxes constantly, you've hit this one too. I run it in Cursor every time I test a new webhook integration.

The fuzzer is in fetchsandbox MCP. It reads final state after each permutation, and reports order_independent=True only when every ordering converges to the same final state.

If you're trying to find webhook ordering bugs, or prove a handler is order-safe, this is the test you didn't know you needed.

Read the full spec for Paddle webhook testing at Paddle sandbox. Learn how fetchsandbox MCP works at fetchsandbox MCP.

We planted a Descope privilege-escalation bug — can your coding agent catch it?

FetchSandbox — Fri, 10 Jul 2026 18:39:39 +0000

We open-sourced a repo with a bug planted on purpose: github.com/fetchsandbox/playground.

The bug is in apps/descope — a small FastAPI "Agent Gateway" where AI agents exchange a Descope access key for a scoped session. The flaw: the exchange endpoint trusts whatever scopes the client requests. A read-only agent key can ask for users:write and get it.

This is the privilege-escalation shape that ships constantly in agentic auth. The token exchange returns 200, the happy path works, and nothing ever compares the granted scope against what the key was actually granted.

The 20-minute test drive

Two tasks, both run from your IDE (Cursor, Claude Code, or any MCP-capable editor). No Descope account, no API keys — the workflows run against FetchSandbox's hosted Descope sandbox over MCP, and each app ships a .mcp.json already wired.

Task 1 — greenfield. apps/descope-onboarding is a tiny notes app with placeholder auth. Ask your agent to add Descope OTP sign-up, but with one constraint: prove the OTP + session flow in the sandbox before writing any code, then propose the diff.

./fetchsandbox I'm adding Descope OTP sign-up to this app — prove the Descope
OTP + session flow in the sandbox before writing any code, then propose the
diff. I'll decide whether to apply.

Task 2 — brownfield. Point the agent at the Agent Gateway and ask it to audit:

./fetchsandbox our agent access-key exchange might be handing out more scope
than the key was granted — audit the descope agentic auth

What a real catch looks like

Static review is not the bar. "This looks vulnerable" is a guess. The bar is:

the escalation is reproduced — read-only key in, users:write session out
the proof shows buggy vs fixed on actual Descope routes (/v1/...)
there's a receipt URL — an openable run trace, not a claim in chat

Auth bugs live in the lifecycle: replayed magic links, expired sessions, JWTs that were decoded instead of verified, scope grants nobody re-checked. Mocks return clean 200s for all of these. That's the gap the playground is designed to expose.

The other planted bugs

If Descope isn't your stack, the repo also has broken Stripe (webhook dedup keyed on the wrong header — events processed 2–3×), Resend (bounces silently dropped, users stay "active"), Clerk (session validation skipped on one endpoint), and a few more. Each app is ~50–150 lines of Python.

The part where you tell us it fell flat

Findings go back as a PR — there's a FINDINGS_TEMPLATE.md in the repo. Paste your agent's session, the receipt URLs, and your honest reaction. Merged PRs show up on your GitHub contribution graph.

We explicitly want the failure reports: MCP wouldn't connect, the agent caught nothing, the proof felt fake. An honest "it didn't work" beats a polite green checkmark.

Start here: TESTING.md.

How to test Descope auth without real keys

FetchSandbox — Wed, 08 Jul 2026 14:04:28 +0000

Title options:

How to test Descope auth without real keys
How to test Descope passwordless auth without a project

Descope is easy to start with, but the part I would test first is not the OTP screen or the magic-link email.

It is what your app does with the session JWT after Descope returns it.

FetchSandbox has a Descope sandbox for the flows most apps wire up first:

email OTP signup
email magic-link sign-in
session refresh
agent/access-key exchange

No real Descope project. No real keys. No inbox setup.

The useful part is that the sandbox models the failure cases too: wrong OTP, expired session, replayed magic link, and the big one: accepting a forged session JWT because the app decoded it instead of verifying it.

Here is the concrete flow I would test.

Send a magic link:

POST /v1/auth/magiclink/signin/email

Verify the one-time token:

POST /v1/auth/magiclink/verify

The sandbox returns the shape your app should care about:

{
  "sessionJwt": "...",
  "refreshJwt": "...",
  "user": {
    "userId": "U2..."
  }
}

Call your protected route with the sessionJwt.

This is where the bug usually lives.

Bad handlers do something like:

const claims = jwt.decode(sessionJwt);
if (claims.role === "admin") allow();

That accepts whatever claims are inside the token. A forged alg: none token can become role=admin if the handler never checks the signature.

The fixed version verifies the session JWT against Descope's JWKS, or uses Descope's SDK validation path. It should reject alg: none, reject bad signatures, cache JWKS, and refresh on kid mismatch.

The sandbox route for this bug is meant to prove that difference:

forged unsigned token -> 401
valid signed session token -> 200

That is the whole point. Do not just test "magic link works." Test that the session you trust is actually verified.

You can also test refresh behavior:

POST /v1/auth/refresh
GET /v1/auth/me

If refresh returns 401, send the user back through sign-in. Do not keep rendering authenticated UI from a decoded-but-unverified token.

Docs are here if you want the exact Descope sandbox flows: https://fetchsandbox.com/docs/descope

Our CI mocked Stripe. Production still broke on step two.

FetchSandbox — Tue, 30 Jun 2026 14:23:58 +0000

Green CI. Broken staging. Sound familiar?

Our pipeline mocked POST /v1/payment_intents and asserted 200. Merge button enabled. First real checkout in staging failed because the handler never stored the PaymentIntent ID — step two of the webhook path read undefined.

Mocks proved shape. Not lifecycle.

What unit tests miss

CI check	What it proves	What it skips
Mocked `fetch()`	Request JSON looks right	Step 2 cannot read step 1's ID
Snapshot tests	Handler code unchanged	Webhook branch never runs
Manual smoke doc	Someone remembers sometimes	Agents ship faster than the checklist

The bug class: IDs do not chain across steps. Webhooks fire on mutations your mock never triggers.

What we run now — MCP locally, same workflow in CI

Before an agent touches payment code, we prove the provider flow in the IDE via FetchSandbox MCP:

quickrun stripe / accept_payment
→ create PaymentIntent
→ confirm / capture path
→ read back state

That is the contract the integration must satisfy — not a paragraph in the PR description.

Then CI runs the same curated workflow headless:

# .github/workflows/api-integration.yml
- name: Prove Stripe integration before deploy
  run: npx fetchsandbox run stripe --all --json

Exit code 0 → workflows passed, merge allowed.

Exit code 1 → named step failed (create_payment_intent, webhook reconcile, etc.) — PR blocked.

No staging dependency. No IP whitelist. No "works on my machine."

Why MCP belongs in the loop

Coding agents edit integration code constantly. Without a runnable proof:

The agent "fixes" the handler and moves on
CI mocks still pass
Production breaks on the path the agent never exercised

MCP gives the agent (and you) a receipt: sandbox timeline, step statuses, real provider-shaped IDs from the run — before opening the PR.

CI is the gate that keeps that proof from regressing on the next agent session.

What a workflow run actually checks

Not "does POST return 200 once."

Create a resource → mutate it → GET the same ID back
Trigger the webhook branch when state changes
Fail with a named step when auth or transitions break

That is integration testing — not HTTP cosplay.

What we stopped doing

Treating mocked unit tests as "integration covered"
Running only manual smoke before deploy
Letting agents merge Stripe/Resend/Clerk changes without a workflow receipt

Try it

Full write-up: API integration testing in CI

MCP + agent workflows: Agent API workflow testing

Install MCP in Cursor/Claude → run a Stripe workflow locally → paste the same npx fetchsandbox run step into Actions.

fetchsandbox.com · MCP setup

Our partners kept breaking on staging. So we gave them production access. (Don't do this.)

FetchSandbox — Thu, 25 Jun 2026 16:43:57 +0000

We whitelisted partners on production with read-only test accounts. That was attempt four — after docs-only onboarding, shared UAT, and IP-whitelisted staging all failed in different ways.

I work on an embed platform. Partners integrate our APIs for payments, identity verification, and onboarding flows. We burned six months on one question: how do partners test their integration before go-live?

Attempt 1: "Just use our docs"

We pointed partners at endpoints, auth, request/response examples. "You're good to go."

They'd hit something we didn't document, open a support ticket, wait two days, lose momentum. A few never came back. One went with a competitor because they "couldn't get a working test setup in a week."

The docs described what should happen. Partners had no way to see what actually happens until they wrote code, deployed, and hoped.

Attempt 2: Internal UAT box

Next: give partners our internal UAT environment. IP whitelisted, shared credentials. We told them "please don't break anything" without irony.

Worked for three weeks.

QA pushed a bad config on a Tuesday. The partner's integration broke. They spent two days debugging their own code before opening a ticket. By the time we figured it out, their engineering lead was cc'ing our VP.

Another team ran load tests on UAT the same week. Partner API calls timed out. Their CTO asked if our platform was "production-ready."

UAT was built for us to break things. We invited external partners into that mess.

Attempt 3: Staging with IP whitelisting

We carved out "partner staging" — same codebase, separate deployment, IP whitelisted per partner. Felt like the grown-up solution.

It wasn't.

IP whitelists became a full-time job. Every new partner meant firewall rules. Home IPs differed from office IPs. One CTO traveling couldn't hit the sandbox from his hotel. We debugged VPN configs at 11pm on a Thursday.

Test data went stale. Partners created orders for products that existed in production but not staging. "Your API returns 404 for product_id XYZ." Correct — nobody seeded staging in three weeks.

Deploys collided with demos. Engineers shipped to staging without checking if partners were testing. A deploy during a partner's live demo call gets brought up in quarterly business reviews.

Cost. Full production mirror for partner testing — database, compute, monitoring, on-call — used maybe 10 hours a week.

Attempt 4: Production access

I wish I was kidding.

Reasoning: staging is flaky, UAT is a disaster, whitelist them on production with read-only test accounts.

The API was stable. Partners were happy. For about a month.

One partner's integration bug created 400 orphaned records in production. Data team spent a weekend cleaning up.

Compliance asked why test payloads with fake PII hit the production database.

Maintenance meant coordinating downtime with partners "just running a few test calls."

We'd built the world's most expensive sandbox: production with IP whitelisting.

What we actually needed

Partners didn't need access to our infrastructure.

They needed an API that looked and acted like ours — same endpoints, schemas, auth patterns — but was completely separate. POST creates a resource. GET retrieves it. State transitions work. Nobody else's deploy breaks it.

Not a mock server — those are stateless; step 2 doesn't know about step 1. Not a shared staging box — that's everybody's problem. A dedicated sandbox generated from the same OpenAPI spec the real API uses.

This is half why I started working on FetchSandbox. Give it an OpenAPI spec, get a stateful sandbox — CRUD, state machines, webhook events, seed data. No infrastructure to maintain.

# Partner gets a sandbox from your spec
npx fetchsandbox generate ./your-api-openapi.yaml

curl https://your-api.fetchsandbox.com/v1/orders \
  -H "api-key: sandbox_abc123"
# → 200 OK, realistic seed data

fetchsandbox run your-api create-and-fulfill-order
# → ✓ Create order — 201
# → ✓ Add line items — 200
# → ✓ Submit for fulfillment — 200
# → ✓ Webhook: order.fulfilled fired

No staging to maintain. No IP whitelists. No production risk. Partners get their own URL, data, and credentials. Your team doesn't touch it.

When a partner asks "does this workflow work?" — they prove it themselves.

The time sink nobody tracks

If you run a partner API, count hours per month on:

Provisioning and maintaining test environments
Debugging "is your sandbox down?" tickets
Managing IP whitelists and VPN access
Re-seeding stale test data
Coordinating deploys around partner testing schedules

At my company: 20–30 hours across engineering and DevOps. For a problem that shouldn't exist.

Open the Stripe sandbox and run a workflow end-to-end — no signup: fetchsandbox.com/playground

Resend webhook reconciliation: map events to the right email record

FetchSandbox — Tue, 23 Jun 2026 20:02:20 +0000

This is a different failure mode from treating email.sent as delivered. That bug is here. This post is about what happens after you store the Resend email ID.

POST /emails returned 200. You saved the Resend email ID on your notification row. A minute later the webhook handler runs.

Then support asks: "Why does the dashboard say delivered when the address bounced?"

The send call worked. The reconciliation layer did not.

The bug is usually a flat status field

Most handlers start like this:

app.post("/webhooks/resend", async (req, res) => {
  const event = req.body;
  const emailId = event.data.email_id;

  const row = await db.notifications.findByResendId(emailId);
  if (!row) return res.sendStatus(200);

  if (event.type === "email.delivered") {
    await db.notifications.update(row.id, { status: "delivered" });
  }
  if (event.type === "email.opened") {
    await db.notifications.update(row.id, { status: "opened" });
  }
  if (event.type === "email.bounced") {
    await db.notifications.update(row.id, { status: "bounced" });
  }

  res.sendStatus(200);
});

Three separate problems hide in that shape:

Lookup failure — webhook arrives before your app finishes persisting the Resend ID, or the ID is stored on the wrong table. Handler returns 200, nothing updates, bug disappears until production.
Last-write-wins — email.opened overwrites delivered. Fine if you track engagement separately. Bad if status drives UX copy or retry logic.
No terminal states — email.bounced should win over delivered when events arrive out of order or retry. A single string field cannot express that without rules.

Separate delivery state from engagement

A pattern that survives production:

const TERMINAL = new Set(["bounced", "complained", "failed"]);

function nextDeliveryState(current, incoming) {
  if (TERMINAL.has(current)) return current;
  if (incoming === "bounced" || incoming === "complained") return incoming;
  if (incoming === "delivered") return current === "sent" ? "delivered" : current;
  if (incoming === "sent") return current ?? "sent";
  return current;
}

app.post("/webhooks/resend", async (req, res) => {
  const { type, data } = req.body;
  const emailId = data.email_id;

  const row = await db.notifications.findByResendId(emailId);
  if (!row) {
    await db.webhookInbox.insert({ provider: "resend", emailId, type, payload: req.body });
    return res.sendStatus(200);
  }

  if (type === "email.opened" || type === "email.clicked") {
    await db.emailEvents.insert({ notificationId: row.id, type });
    return res.sendStatus(200);
  }

  const next = nextDeliveryState(row.deliveryStatus, type.replace("email.", ""));
  await db.notifications.update(row.id, {
    deliveryStatus: next,
    lastEventType: type,
    lastEventAt: new Date(),
  });

  res.sendStatus(200);
});

Notes that matter in real apps:

email.opened is not delivery. Log it separately unless your product truly treats opens as delivery confirmation.
Store unmatched webhooks. If the event arrives before the send transaction commits, you need a replay path — not a silent 200.
Idempotency key = Resend email ID + event type. Retries are normal.

The test most teams skip

Unit tests mock one webhook payload. That proves parsing, not reconciliation.

The integration test that catches the bug:

create internal notification row
POST /emails and persist Resend email ID on that row
deliver email.sent, then email.delivered
deliver email.opened and assert delivery status did not regress
deliver email.bounced on a second send and assert terminal state wins

Step 4 and 5 are where flat status fields break.

Receipts

Real Resend send workflow run against FetchSandbox, captured 2026-06-07:

POST /emails                                       → 200
  id: 4d9acb24-5913-4e02-8b1a-a18ed10a6fad

GET  /emails/4d9acb24-5913-4e02-8b1a-a18ed10a6fad  → 200

webhook events verified:
  email.sent
  email.delivered

Full timeline: fetchsandbox.com/runs/8e4fe8e9f9?flow=run_4062510e-268e-451f-b5f4-e8844eef42d5

That run proves the provider lifecycle through email.delivered. Your app still needs its own test for mapping those events back to the correct internal row and for terminal bounce/complaint handling.

Runnable preflight (optional)

Before wiring production webhooks, I run the send workflow from Cursor with FetchSandbox MCP:

validate resend email workflow with fetchsandbox

Workflow page: fetchsandbox.com/docs/resend

Resend sandbox page (mock API + webhook replay): fetchsandbox.com/resend

That does not replace Resend's real domain, API key, or production webhook endpoint. It just makes the lifecycle obvious before you commit your local state machine.

Test AgentMail multi-tenant webhooks before they leak across tenants

FetchSandbox — Tue, 16 Jun 2026 17:16:05 +0000

The dangerous AgentMail webhook bug starts with a response that looks completely fine.

POST /v0/webhooks -> 200 OK

The response body contains the webhook object. It includes your URL, event types, and usually enough fields to make the integration feel done.

But in a multi-tenant app, the field that matters is inbox_ids. If that array is silently dropped, your webhook is no longer scoped to the tenant inbox you meant to subscribe. It is subscribed broadly, and every tenant's message events can start flooding the same endpoint.

That is not a noisy failure. It is worse. The subscribe call worked. Your endpoint receives events. The logs look active. The leak is in the scope.

If your handler routes by webhook ID before it checks the inbox ID, the wrong customer can look like a valid event. That is the kind of bug that passes observability and fails privacy.

Why unit tests miss it

Most unit tests stub the webhook create call by echoing back exactly what the app sent.

request.inbox_ids -> response.inbox_ids

That proves your client serialized the payload. It does not prove AgentMail persisted the scope.

The test passes. CI passes. The code ships. Then production receives events for inboxes that do not belong to the tenant that configured the webhook.

The annoying part is that the local test is not obviously wrong. It has the right endpoint, the right payload shape, and the right assertion against the create response. It just trusts the wrong copy of the object.

The pattern is reconcile-after-write

The fix is small enough to name:

POST -> GET -> diff

Treat the POST response as untrusted. It may be your request echoed back. Treat the later GET as the provider state. That is what AgentMail actually stored.

The diff is the bug.

const sentInboxIds = ["inbox_tenant_123"];
const created = await agentmail.post("/v0/webhooks", {
  url: "https://app.example.com/webhooks/agentmail",
  event_types: ["message.delivered", "message.bounced"],
  inbox_ids: sentInboxIds,
});
const stored = await agentmail.get(`/v0/webhooks/${created.id}`);
const got = [...(stored.inbox_ids ?? [])].sort();
const want = [...sentInboxIds].sort();
if (JSON.stringify(got) !== JSON.stringify(want)) {
  throw new Error("AgentMail webhook scope drifted");
}

That is the whole check. It is not a bigger mock. It is a read after the write.

Why the agent caught it

The reason a coding agent caught this where the unit test did not is not that it was smarter about webhooks. It ran a better-shaped test.

AgentMail has a curated webhook_lifecycle_create_read_delete workflow in FetchSandbox. When the agent ran it through the FetchSandbox MCP server, the workflow forced the boring step people skip:

create inbox
subscribe webhook scoped to that inbox
read webhook back
compare inbox_ids
delete webhook

The workflow shape matches the bug shape. A stubbed unit test checks what your app meant to send. The workflow checks what the provider kept.

This is not only AgentMail

The same bug appears anywhere a control-plane API returns an object that looks like the request you sent:

IAM policies
ACL rules
draft resources
webhook subscriptions
tenant-scoped notification settings

If a create call returns 200, but the security or tenancy boundary matters, do not stop at the create response.

Read it back. Diff the fields that protect the boundary. Fail the test before the provider starts sending another tenant's events to your endpoint.

Run the brownfield demo

The brownfield-agentmail-demo repo shows this end to end if you want to run it against the workflow yourself.

The important part is not AgentMail-specific. The habit is: after a write that configures scope, reconcile what the provider persisted.

A webhook that passes create tests can still be the webhook that leaks across tenants.

FetchSandbox's AgentMail workflow docs include the create, read, and teardown path.

Clerk webhook replay can create duplicate users unless your sync is idempotent

FetchSandbox — Mon, 15 Jun 2026 13:57:02 +0000

The login worked. The webhook worked. Then the same event arrived again.

Clerk webhooks are easy to treat like a notification:

user.created arrived
create a local user
done

That works until the event is retried, replayed, or delivered after another part of your app already created the user.

Then your app has two local rows for the same Clerk user. Or one row has the right email and the other has the right metadata. Or your onboarding job runs twice and sends the same welcome email twice.

The bug is not "Clerk webhooks are unreliable." Retrying delivery is normal webhook behavior.

The bug is trusting delivery count instead of provider state.

The narrow fix

Your webhook handler should not mean:

on user.created -> insert user

It should mean:

on user.created -> upsert by clerk_user_id

The important field is the stable provider ID, not the local row ID and not the email address.

await db.user.upsert({
  where: { clerkUserId: event.data.id },
  update: {
    email: event.data.email_addresses?.[0]?.email_address,
    firstName: event.data.first_name,
    lastName: event.data.last_name,
    clerkUpdatedAt: new Date(event.data.updated_at),
  },
  create: {
    clerkUserId: event.data.id,
    email: event.data.email_addresses?.[0]?.email_address,
    firstName: event.data.first_name,
    lastName: event.data.last_name,
    clerkUpdatedAt: new Date(event.data.updated_at),
  },
});

That one constraint does most of the work:

UNIQUE(clerk_user_id)

Now a replay updates the same user instead of creating a second one.

The part people skip

The handler can still be wrong even after the upsert.

If your app grants access, creates a workspace, starts a trial, or sends email inside the same handler, those side effects need their own idempotency rule too.

For example:

workspace owner = clerk_user_id
welcome email key = clerk_event_id
trial grant key = clerk_user_id + plan_id

The user row is only one piece of local state.

How I would test it

I would test the same user.created event twice.

First delivery:

local user count: 0 -> 1
workspace count: 0 -> 1

Replay:

local user count: 1 -> 1
workspace count: 1 -> 1

That is the whole test.

FetchSandbox has a Clerk sandbox and runnable Clerk workflow docs for this exact kind of replay check. It is also where a stateful sandbox is more useful than a static mock response: the second delivery is the thing you need to observe.

Takeaway

Do not write Clerk webhook handlers as if events happen once.

Write them as reconciliation:

given this Clerk user ID,
what should my app state be now?

If the answer changes when the same event arrives twice, the bug is already there. Production is just waiting to replay it.

DEV Community: FetchSandbox

undefined === undefined — the auth bypass your AI wrote into your checkout

What the check actually did

The real check

Why agents write it this way

Fail closed, and prove it

What I changed in the habit

Green passed. The fix granted zero seats.

What the check actually asserted

The real invariant

Why agents fail this way

Positive control, negative control

What I changed in the habit

You can't code-review a webhook retry

Reading can't catch it

CI passed for the same reason

The review that actually works

Why Paddle's subscription.activated arrives before subscription.created

What actually happened

Why this happens

The fix

Testing this is the real problem

The takeaway

Webhook handlers that silently return 402: the event ordering bug most test suites miss

We planted a Descope privilege-escalation bug — can your coding agent catch it?

The 20-minute test drive

What a real catch looks like

The other planted bugs

The part where you tell us it fell flat

How to test Descope auth without real keys

Our CI mocked Stripe. Production still broke on step two.

What unit tests miss

What we run now — MCP locally, same workflow in CI

Why MCP belongs in the loop

What a workflow run actually checks

What we stopped doing

Try it

Our partners kept breaking on staging. So we gave them production access. (Don't do this.)

Attempt 1: "Just use our docs"

Attempt 2: Internal UAT box

Attempt 3: Staging with IP whitelisting

Attempt 4: Production access

What we actually needed

The time sink nobody tracks

Resend webhook reconciliation: map events to the right email record

The bug is usually a flat status field

Separate delivery state from engagement

The test most teams skip

Receipts

Runnable preflight (optional)

What to read next

Test AgentMail multi-tenant webhooks before they leak across tenants

Why unit tests miss it

The pattern is reconcile-after-write

Why the agent caught it

This is not only AgentMail

Run the brownfield demo

Clerk webhook replay can create duplicate users unless your sync is idempotent

The narrow fix

The part people skip

How I would test it

Takeaway