DEV Community: FetchSandbox

Lovable Builders Break One Action On Purpose. Nobody Named It.

FetchSandbox — Thu, 01 Oct 2026 16:46:15 +0000

In 47 posts, three builders describe failure testing before launch.

Lovable builders are already testing failures by hand. I read 47 posts from r/lovable and found three that point at the same habit: break an important action on purpose before launch.

What breaks after the happy path

The posts describe a familiar gap. A webhook route looks fine, then takes days to debug when it fails. A new app's error logs make sense to the person who wrote the backend, but not to the person trying to help a customer. From the outside, a failed payment or missing email can look like a confusing screen, not a broken integration.

Generated apps usually exercise the success case while they are being built. Production adds a second webhook delivery, a payment that succeeds while the app's membership row stays free, or an email request that returns 200 but later bounces. None of those is answered by asking whether the page loaded.

Break the action that can fail quietly

Choose the action that would cost you money or trust if it failed without an obvious error—often checkout or its confirmation email. Force a failure and inspect what the customer sees and what the database stores.

Two useful first checks:

Deliver the same webhook twice. Does the app provision twice?
Let payment succeed, then make the follow-up write fail. Does the customer remain marked free? That mismatch has its own failure pattern.

People rebuild these checks by hand after an agent rewrites the backend. The hosted FetchSandbox MCP can start an app-level check: configure the returned twins, arm a returned probe with the session ID, exercise the reachable app, and submit the session ID, run ID, and probe. A green provider quickrun alone does not test the Lovable app.

When code moves into a repository, carry applicable deterministic checks into CI and attach the receipt to the change. A reviewer can then inspect what ran after the builder chat is gone. Anything the run did not measure should remain unmeasured.

The full 47-post write-up lists the titles and limits. This is a small sample biased toward people describing problems, not a measure of how often failures happen. One record was an unrelated game ad; I did not interview the posters. The useful signal is narrower: builders are already asking how to break the important path before users find it.

Replit's Green Twin Run Never Hit Its App

FetchSandbox — Thu, 01 Oct 2026 16:38:36 +0000

A green quickrun can test the twin while leaving your app untested.

Replit says the Paddle checkout works. The confirmation email appears. A green quickrun can still leave the app untested.

That is because the quickrun exercises a FetchSandbox workflow against a provider twin. It does not call the Replit checkout, webhook handler, database, or email path. You can have a healthy twin and an app that still grants access after a declined payment.

Make the app part of the test

Connect FetchSandbox through Replit's hosted MCP URL, https://fetchsandbox.com/mcp/v1, with a Bearer key. Then ask the agent to call validate_integration for Paddle and Resend with the app's reachable URL. That starts an app-level session and returns the twin configuration and available probes for that run.

The Replit app must actually use those temporary twin hosts from its development or disposable test configuration. Keep live provider keys out of this run. If the test needs to deliver a webhook, the app's webhook route must be reachable from FetchSandbox; a private preview that redirects to a login page is not reachable evidence.

Next, arm one probe the session returned, exercise the matching action in the app, and submit that session_id, the run_id returned by arming, and the probe back to FetchSandbox. That links the provider request to what the running app did. A green quickrun is useful for checking the twin workflow, but it is not a substitute for this path.

Inspect the run's delivery attempts and checkpoints. A redirect, signature rejection, or missing app observation means the relevant check did not run; repeating the same setup does not turn it into a pass. Fix reachability or configuration first, then start a fresh run when its temporary session has expired.

Try this instruction in Replit:

Use FetchSandbox MCP validate_integration for Paddle and Resend
against this app's reachable test URL. Configure only the returned
temporary twins in the test environment. Show the session_id and probes.
Arm one with session_id, exercise the app flow, then submit session_id,
run_id, and probe. Do not call quickrun an app test, and report every unmeasured
check plainly.

If the app unlocks a membership after a failed payment, inspect the membership state—not just the HTTP 200. If a receipt rule is unmeasured, the run did not establish that behavior. A useful MCP workflow makes that difference visible; it cannot turn missing evidence into a pass.

When the code moves into a repository, carry the relevant deterministic checks into CI and attach the run receipt to the change. The agent-to-pipeline workflow makes the result reviewable after the Replit chat is gone.

The rule is simple: a green twin workflow proves the twin path; only observed app traffic and measured app behavior prove the app checks that ran.

Lovable's MCP Form Has a None Option. Don't Pick It.

FetchSandbox — Tue, 22 Sep 2026 15:58:09 +0000

Lovable will let you add a custom MCP server with No authentication.

That option is a trap for FetchSandbox. The hosted endpoint at https://fetchsandbox.com/mcp/v1 forwards Authorization to the backend. Empty bearer is a 401. The 401's job is to send you to /keys.

Get the key first

Do not paste it into Lovable chat.

Then the form

Connectors → + → MCP server:

Server name: FetchSandbox
Server URL: https://fetchsandbox.com/mcp/v1
Authentication: Bearer token or API key
Token: the secret you just copied

Leave OAuth alone. FetchSandbox does not complete that dance from this form.

Chat connectors stay on your account. They are not in the published app.

Then make it misbehave

Spin up a Stripe test environment and run a payment through it.
Arm webhook_retries and run checkout again.

A 200 on the first charge is the easy part. The question is whether the same event id credits twice.

The screenshots, the JWT 401 path, and the "paid but still free" read-back are in the canonical guide.

Deterministic Evals for Coding Agents: Prove the Fix

FetchSandbox — Tue, 15 Sep 2026 12:22:28 +0000

Your coding agent says it fixed the retry bug. The diff looks reasonable, its new test passes, and an LLM reviewer gives the patch a high score. None of those results proves that a duplicate webhook can no longer charge the customer twice.

The missing piece is a deterministic eval that makes the failure happen on the old code, runs the same failure against the proposed fix, and lets measured behavior decide the verdict.

What is a deterministic eval engine for coding agents?

A deterministic eval engine for coding agents executes a defined invariant against real code and calculates the result with fixed logic. Given the same code, scenario, and observations, it returns the same verdict. A model can help discover a bug or propose a test, but it does not get to turn the final run green.

For a bug fix, the useful evaluation is differential:

Start with the current, still-broken working tree.
Apply the proposed patch to a separate copy.
Inject the same provider failure into both copies.
Run the same executable probe against both.
Require the old code to violate the invariant.
Require the patched code to hold the invariant.
Record the observations in a receipt.

FetchSandbox implements this pattern for third-party API integrations. It is a verification layer between the coding agent's claim and the pull request.

Why is an LLM judge not enough to verify a code fix?

An LLM judge is useful when the requirement is subjective: Is the explanation clear? Is the design unnecessarily complex? Does the patch appear maintainable?

An integration invariant is usually not subjective.

One payment event should create one entitlement.
A replayed webhook should not repeat a charge.
A provider 409 should not be swallowed and reported as success.
A lost response should not make a retry create the resource twice.
A stale event should not overwrite newer local state.

For those properties, a probability or prose assessment is weaker than an observation. The relevant question is not whether another model likes the patch. It is whether the required state held after the failure actually ran.

Anthropic's guidance on evaluating AI agents makes the same grader distinction: use deterministic graders where correctness can be checked by code, and reserve model-based graders for criteria that require judgment.

That separation matters in production. If generation and final grading are both model decisions, the verification layer inherits model variance. If execution produces a stable signal and pure logic maps that signal to a verdict, CI can rely on it.

How does the exit-code flip decide whether a fix is proven?

FetchSandbox probes use a small exit-code contract:

0 means the invariant HELD
1 means the invariant was VIOLATED
2 or another harness error means INCONCLUSIVE
a probe that did not run is also INCONCLUSIVE

Only one measured transition can produce a green verdict:

buggy tree   1  VIOLATED
fixed tree   0  HELD
result          PROVEN
green_allowed   true

Every other result declines or rejects the proof:

0 → 0: the supposed bug did not reproduce, so the patch is not proven
1 → 1: the bug reproduced, but the fix is incomplete
either side inconclusive: the engine could not establish the claim
self-reported before-and-after results: useful context, but never independently proven

This gate is deliberately boring. It is a pure function of the two measured runs. Reference simulations, agent confidence, a persuasive explanation, or a passing unrelated test cannot change green_allowed to true.

Why must the same failure run on old and fixed code?

Running only the patched code proves less than it appears to.

Suppose a coding agent adds a test for a duplicate webhook and the test passes. Several explanations are still possible:

the test never triggered a duplicate delivery
the handler never booted
the mock skipped the provider behavior
the assertion checked only the HTTP response
the patch suppressed every side effect, including the one the customer paid for
the bug was never present in the tested path

The old-code run is the negative control. It demonstrates that the scenario can detect the claimed bug in this repository. The fixed-code run then demonstrates that the same detector no longer finds it after the proposed change.

Without the first leg, “pass” can mean “the check is blind.” Without the second, there is no fix. The pair is what turns a test result into differential evidence.

How does FetchSandbox inject integration failures?

FetchSandbox runs integration code against a stateful sandbox of the services it talks to. A scenario can deliberately introduce behavior that happy-path mocks often omit, including duplicate webhook delivery, 409 conflict responses, retries, and observable repeated side effects.

Consider a payment webhook handler that deduplicates on a delivery ID instead of the provider's stable event ID. A retry arrives with a new delivery ID, so the application treats one event as two and grants or charges twice.

The executable invariant is not:

The handler returned 200.

It is:

One provider event creates one business side effect,
even when delivery is attempted more than once.

The probe drives the duplicate delivery, observes the application state, and exits 1 if the side effect accumulates. The proposed patch is applied to a copy, the same delivery sequence runs again, and the probe exits 0 only when the invariant holds.

FetchSandbox also has a zero-effect backstop for this class of proof. A patch that stops the duplicate by granting or charging nothing does not get credit for correctness.

Is there an LLM anywhere in the FetchSandbox eval path?

The final verdict is not an LLM opinion. Once both probe runs complete, fixed logic classifies their exit codes and decides whether the transition qualifies for green.

The scenario tier is a separate concern:

curated scenarios use reviewed, hand-authored probes
declared scenarios use an integration's explicit invariant definition
a novel bug class can use an LLM to propose a candidate probe

A generated probe is not trusted because a model wrote it. It must first reproduce the failure on the buggy code, execute against both trees, state the invariant it checked, and survive qualification before it can be reused. If FetchSandbox cannot produce a probe that runs and reproduces the bug, the result stays unproven.

This is the important boundary: models may help propose what to execute; models do not decide whether the measured execution passed.

What does a coding agent send to `prove_fix`?

The prove_fix MCP tool needs the still-broken project and the proposed unified diff. Call it before applying the patch to the working tree.

find_bugs
  → fix_bug returns a proposed diff
  → prove_fix receives broken tree + diff
  → FetchSandbox applies diff to a copy
  → same scenario runs against both trees
  → deterministic gate returns proven, rejected, or unproven

A successful response includes machine-readable fields such as:

{
  "state": "proven",
  "green_allowed": true,
  "reproduced": true,
  "verified": true,
  "receipt_url": "https://fetchsandbox.com/runs/fix-...?flow=..."
}

If the agent has already applied the patch, the original failing state is gone from the workspace. Reverting mentally or writing a new fixed-only test is not equivalent. The tool needs the real broken tree to establish the first half of the proof.

What is in the proof receipt?

The receipt is the review artifact for the run. It records the scenario, the before-and-after outcome, the invariant, and the measured verdict at a shareable URL.

Here is a measured FetchSandbox proof receipt from a Paddle entitlement bug. The first proposed patch was rejected as incomplete. A later patch reproduced the bug on the old tree and held the invariant on the fixed tree, producing the required 1 → 0 transition.

A reviewer can paste the receipt into the pull request:

## Integration proof

- Failure: duplicate webhook repeats a paid side effect
- Invariant: one provider event creates one side effect
- Before patch: VIOLATED
- After patch: HELD
- Verdict: proven
- Receipt: https://fetchsandbox.com/runs/fix-...?flow=...

The receipt does not replace review. It removes one factual question from review: whether the named integration failure reproduced before the patch and stopped after it.

How does the proof fit into PR review and CI/CD?

Use the proof receipt during pull-request review, then keep the corresponding provider workflow in CI.

The two checks serve different moments:

prove_fix evaluates a proposed patch against a reproduced bug and returns the before-and-after receipt
fetchsandbox run --all --json runs the configured integration workflows in a pipeline and exits non-zero when a workflow fails

The current shipped workflow is to paste the prove_fix receipt into the PR manually. FetchSandbox does not yet post GitHub comments automatically. In CI, store the JSON workflow output as an artifact and make the job a required check when its coverage matches the branch's risk.

This creates a useful chain of evidence:

agent proposes code
  → sandbox reproduces the failure
  → patch flips VIOLATED to HELD
  → receipt travels with the PR
  → workflow remains gated in CI/CD

The agent remains fast. The reviewer gets a falsifiable result instead of a longer assurance.

Does FetchSandbox evaluate every kind of coding task?

No. FetchSandbox is focused on code that integrates with third-party APIs and on failure modes its scenarios and probes can observe. The measured real-code runner currently boots Node.js and Python projects.

It does not prove subjective code quality, migration safety, authorization design, or every requirement in a pull request. Those still need static checks, ordinary test suites, security review, and human judgment.

It also does not turn missing coverage into success. If the app cannot boot, the scenario cannot reproduce the bug, or the probe cannot judge the outcome, the verdict remains unproven.

That refusal is part of the product. A deterministic eval engine is valuable not because it always says yes, but because nothing except the required measured transition can make it say yes.

How do I use FetchSandbox from a coding agent?

Connect FetchSandbox MCP to Claude Code, Cursor, Claude Desktop, Codex, or another MCP-compatible client. The agent can discover integrations, run stateful workflows, inject failures, investigate a bug, propose a patch, and call prove_fix before editing the working tree.

For the broader setup, see agent API workflow testing and API integration testing in CI.

Questions about deterministic coding-agent evals

What is the difference between a coding-agent benchmark and a fix proof?

A benchmark compares agent performance across a task set. A fix proof evaluates one concrete claim in one repository: the named failure occurred before this patch and stopped after it under the same executable scenario.

Can the same coding agent generate the fix and request the proof?

Yes. Independence comes from the external scenario, real execution, and deterministic gate. The agent can request the run, but it cannot set green_allowed.

Why not trust a test that the coding agent wrote?

Agent-written tests can be useful, but they can repeat the same mistaken assumption as the patch. The stronger pattern includes a negative control: the probe must catch the bug on the old code before its pass on the new code counts.

What happens if the old code passes?

The result is not_reproduced, not proven. A 0 → 0 result shows that the probe did not establish the claimed before-and-after change.

What happens if the runner or probe crashes?

The run is inconclusive and green is blocked. A harness error is not evidence that the application violated or held the business invariant.

Can an LLM-generated probe produce a green result?

It can only contribute a candidate executable check. The candidate must run against real code, reproduce the old failure, and observe the fixed invariant. The deterministic gate, not the model, decides the result.

RevenueCat Returned 200. The Wallet Still Went Backward.

FetchSandbox — Tue, 15 Sep 2026 12:19:29 +0000

A paying customer cancelled, then recovered. RevenueCat sent both events. The handler verified the signatures and returned 200 twice.

The wallet ended on the cancelled state.

The older EXPIRATION had been retried after the newer RENEWAL. Arrival order was not event order, and the last write won.

Someone in an Indie Hackers thread put the same bug in one sentence:

My RevenueCat webhooks write to a wallet balance, and the real failures were never bad code, just two events landing out of order.

What the handler assumed

The code looked ordinary:

app.post("/webhooks/revenuecat", async (req, res) => {
  const event = verifyRevenueCat(req);

  if (event.type === "RENEWAL") {
    await db.users.update(event.app_user_id, { credits: 100, entitled: true });
  }

  if (event.type === "EXPIRATION") {
    await db.users.update(event.app_user_id, { credits: 0, entitled: false });
  }

  res.sendStatus(200);
});

It treated each POST as current truth. That only works if HTTP arrival order matches the subscription lifecycle.

RevenueCat does not promise that. Failed deliveries retry after about 5, 10, 20, 40, and 80 minutes. A retry reuses event.id and event_timestamp_ms. It recomputes the signature timestamp t for that HTTP attempt.

So this is a legal sequence:

10:00  EXPIRATION happens
10:01  RENEWAL recovery happens
10:02  RENEWAL webhook arrives     → credits = 100
11:22  EXPIRATION retry arrives    → credits = 0

Both signatures can be valid. Both handlers can return 200. The customer is locked out because the stale write landed last.

The three clocks that disagree

Do not sort RevenueCat events by whichever timestamp is closest to the request.

Field	What it actually is
`purchased_at_ms` / expiration dates	When the store action happened
`event_timestamp_ms`	When RevenueCat generated the event
signature `t`	When this delivery attempt was signed

t is the worst field to use as business order. An 80-minute retry of an old event gets a fresh signature time, so it looks newer than events that genuinely happened after it.

event.id is the idempotency key. Use it to refuse a duplicate delivery. It does not tell you whether this event is still current.

The doorbell pattern

The Indie Hackers replies kept circling the same fix: stop treating the payload as a command.

Store the event. Return 200. Then ask RevenueCat what is true now.

await db.processedEvents.insert({ id: event.id }).catch(duplicate => {
  return res.sendStatus(200);
});

const entitlements = await revenuecat.customers.activeEntitlements(
  event.app_user_id,
);

await db.users.update(event.app_user_id, {
  entitled: entitlements.some((e) => e.id === "premium"),
});

A late retry of an old event then triggers the same current-state read. It cannot roll the wallet backward just because it arrived last.

Wallet credits still need a ledger. If one event means one additive grant, insert event.id under a unique constraint in the same transaction as the credit row. Idempotency answers “did I already process this id?” Ordering answers “is this still the latest truth?” Those are different questions.

The test happy-path fixtures hide

Most RevenueCat webhook tests send one event, assert 200, and stop. A webhook sandbox is only useful here if it can replay the same signed events in more than one order.

The useful fixture sends the same event set in more than one order:

RENEWAL → EXPIRATION
EXPIRATION → RENEWAL
RENEWAL, then the same RENEWAL again
old EXPIRATION retried after a newer RENEWAL

For every permutation, read the application state, not the HTTP status:

final entitlement == RevenueCat active entitlements
each event.id creates at most one side effect
a stale delivery cannot overwrite newer state
the handler did not “fix” order by ignoring every event

That last line is the trap. A patch that writes nothing is perfectly order-independent and completely wrong.

Another comment in that thread asked for the same honesty in the receipt: if order dependence between two events was never tested, say so instead of reporting green.

Prove it from the coding agent, then keep it in CI

Connect FetchSandbox MCP in Cursor or Claude Code. The RevenueCat workflows do not stop at “we received the webhook.” webhook_event_verified and entitlement_verified_after_purchase both read /active_entitlements after the write.

Ask the agent:

Audit my RevenueCat webhook for out-of-order wallet updates.
Keep the still-broken tree. Propose a diff. Call prove_fix before
writing it to disk. The probe must fail on the old code when an
older EXPIRATION arrives after a newer RENEWAL, and hold on the
patched code. Paste the receipt.

Green is only allowed on a measured flip:

buggy tree  → terminal state diverged  → exit 1
fixed tree  → all tested orders agree  → exit 0

If the probe cannot reproduce the order bug, the result stays unproven. A model saying “this looks idempotent now” is not the gate.

Then keep the same fixtures in CI so a later change does not bring the last-write-wins handler back:

- name: Run API integration workflows
  run: |
    npx fetchsandbox run "$FETCHSANDBOX_ID" --all --json \
      > fetchsandbox-workflows.json

Paste the prove_fix receipt into the pull request by hand. Automatic GitHub comments are not a shipped FetchSandbox feature.

## RevenueCat order proof

- event set: RENEWAL, EXPIRATION
- duplicate event.id: exercised
- before: wallet followed arrival order
- after: wallet matched active entitlements
- not verified: two concurrent deliveries for the same customer
- receipt: https://fetchsandbox.com/runs/...

The full timestamp map, retry schedule, and checklist live in the canonical guide.

Your Paddle Webhook Is Signed. The Seat Count Is Not.

FetchSandbox — Sat, 12 Sep 2026 12:49:17 +0000

The webhook passed signature verification. Paddle said the transaction was complete. The customer paid for 5 seats.

The database granted 500.

Nothing forged the webhook. The dangerous value had entered earlier, through customData in browser checkout code. Paddle stored it, included it in transaction.completed, and correctly signed the entire payload.

The handler confused authentic delivery with authoritative data.

The two quantities inside one valid event

Paddle Checkout can receive custom metadata from frontend code:

Paddle.Checkout.open({
  items: [
    {
      priceId: "pri_seat",
      quantity: 5,
    },
  ],
  customData: {
    workspaceId: "ws_123",
    seats: 500,
  },
});

When the transaction completes, the webhook may carry both values:

{
  "event_type": "transaction.completed",
  "data": {
    "custom_data": {
      "workspaceId": "ws_123",
      "seats": 500
    },
    "items": [
      {
        "price": { "id": "pri_seat" },
        "quantity": 5
      }
    ]
  }
}

Both are inside the body protected by Paddle-Signature.

But the fields answer different questions:

custom_data.seats says what checkout code attached
items[0].quantity says what the completed transaction billed

The signature proves Paddle sent those bytes. It does not turn browser-originated metadata into a billing fact.

The handler that looked secure

The vulnerable code was not missing signature verification:

const event = paddle.webhooks.unmarshal(
  rawBody,
  process.env.PADDLE_WEBHOOK_SECRET!,
  signature,
);

if (event.eventType === "transaction.completed") {
  const seats = Number(event.data.customData?.seats ?? 1);

  await grantSeats(
    event.data.customData?.workspaceId,
    seats,
  );
}

It checked the signature, waited for the completed event, and wrote the entitlement on the server.

Then it trusted the wrong field.

The same problem exists with workspaceId. If browser code supplies the tenant identifier and the webhook handler treats it as authorization, payment for one checkout can be routed to an account the server never approved.

What the fixed trust boundary looks like

Use custom data for correlation, not authority.

The safer flow is:

Create a purchase intent on your server.
Store the intended workspace against an opaque checkout reference.
Put only that reference in Paddle customData.
On transaction.completed, resolve the reference on your server.
Find the expected, server-known price.id inside data.items[].
Grant the quantity on that paid line item.
Deduplicate on Paddle's event_id.

The quantity extraction should be explicit:

const seatItem = event.data.items.find(
  (item) =>
    item.price?.id === process.env.PADDLE_SEAT_PRICE_ID,
);

const seats = seatItem?.quantity;

if (
  !Number.isSafeInteger(seats) ||
  seats < 1 ||
  seats > 10_000
) {
  throw new Error("Invalid billed seat quantity");
}

That upper bound is an application decision. The important part is that the entitlement comes from the completed, paid line item rather than an arbitrary metadata key.

Then resolve ownership separately:

const checkoutRef = event.data.customData?.checkoutRef;

const purchase = await db.purchaseIntent.findUnique({
  where: { checkoutRef },
});

if (!purchase) {
  throw new Error("Unknown checkout reference");
}

await applyEntitlement({
  workspaceId: purchase.workspaceId,
  eventId: event.eventId,
  transactionId: event.data.id,
  seats,
});

Payment facts, correlation metadata, and tenant authorization are now three separate concerns.

The test that exposes the bug

A happy-path webhook fixture usually gives both fields the same value:

custom_data.seats      = 5
items[0].quantity      = 5

The vulnerable and fixed handlers both pass.

The useful test makes the trust boundary disagree:

Paddle-Signature       = valid
custom_data.seats      = 500
items[0].quantity      = 5
expected grant         = 5

Do not stop at response.status === 200. Read the entitlement your handler wrote.

The complete invariant is:

5 units billed
500 claimed in client metadata
5 seats granted
same event delivered again
still 5 seats granted

That last line matters because Paddle uses at-least-once webhook delivery. Fixing the quantity source while leaving the grant non-idempotent still doubles access on a retry.

Run it from the coding agent

FetchSandbox has a deterministic paddle_seat_quantity probe for this exact mismatch. It sends a freshly signed transaction.completed event to the application's real handler and observes how many seats the application grants.

With FetchSandbox MCP connected to Cursor or Claude Code:

./fetchsandbox audit my Paddle webhook for client-controlled seat
quantity. Propose the fix, then call prove_fix before applying it.
Show whether the buggy tree grants 500 while the fixed tree grants
the 5 units actually billed. Return the proof receipt.

The same probe runs against both trees:

buggy tree  → granted=500, billed=5 → VIOLATED
fixed tree  → granted=5,   billed=5 → HELD

If the probe cannot load the handler or measure the grant, the result is inconclusive. It does not turn “could not run” into green.

Here is a measured Paddle seat-quantity receipt.

Carry the proof into review and CI

Attach the receipt to the pull request with the exact invariant:

## Paddle entitlement proof

- signature: valid
- paid quantity: 5
- client metadata claim: 500
- before patch: granted 500
- after patch: granted 5
- receipt: https://fetchsandbox.com/runs/fix-...?flow=...

That receipt proves the proposed diff against the mismatched-data failure. Keep the broader provider workflows running in CI as a separate regression gate:

- name: Run API integration workflows
  run: |
    npx fetchsandbox run "$FETCHSANDBOX_ID" --all --json \
      > fetchsandbox-workflows.json

The PR receipt and CI check have different jobs. The receipt shows the bug existed and the proposed fix changed the measured behavior. CI keeps the provider workflows from drifting as more code lands.

Signed does not mean trusted

Verify every Paddle webhook signature. That is non-negotiable.

Then make a second decision for every field: where did this value originate, and is it authoritative for the action I am about to take?

For seat provisioning:

custom_data can correlate the transaction
a server-side record decides the workspace
paid items[] decide the quantity
event_id makes processing idempotent

A valid signature can carry an unsafe instruction. The verifier has to test both the cryptography and the business trust boundary.

The full implementation guide is on FetchSandbox.

AI Code Is Cheap. Your Pull Request Still Needs Proof.

FetchSandbox — Tue, 08 Sep 2026 20:59:59 +0000

Your coding agent opens a plausible pull request before lunch. The tests it wrote pass, the diff reads cleanly, and the reviewer is left with the expensive question: did this change survive the provider failure it claims to handle?

Writing the integration is now the cheap step. Establishing that the integration behaves correctly under retries, lost responses, stale events, webhooks, and partial failure is still the hard step.

How should you verify AI-generated code before merging it?

Verify AI-generated integration code with an independent runtime check, not only the agent's explanation or the tests it wrote. Reproduce the provider failure against the still-broken code, apply the proposed diff to a copy, run the same failure again, and require a measured broken-to-fixed result before merge.

The output should be a shareable receipt that records:

the failure scenario that ran
whether the bug reproduced before the patch
whether the invariant held after the patch
the requests, responses, and observed side effects
an explicit proven, rejected, or unproven verdict

That receipt changes the pull-request conversation from “the agent says this works” to “here is what ran, and here is what changed.”

Why is reviewing AI-written code becoming the bottleneck?

AI coding agents can produce more diffs than a team can carefully inspect. Faster generation increases review pressure because plausible code still has to be checked against the real behavior of every external system it touches.

Static review remains useful. It can catch an unsafe query, a missing authorization check, or an obviously incorrect SDK call. But many integration bugs do not live on one suspicious line:

the same webhook is delivered twice
a successful response disappears after the provider commits the write
a stale event arrives after a newer event
the first request creates an ID that the second request never persists
a handler returns 200 but leaves local state unchanged

The diff can look reasonable while the lifecycle is still broken.

Most current writing about AI code verification says the same broad thing: define acceptance criteria, run tests, and keep humans in the loop. That advice is correct, but it usually stops before the reviewer gets an inspectable artifact showing the failure on the actual code.

The missing layer is runtime evidence attached to the change.

Why is a green test written by the same agent not enough?

A test written by the same agent can repeat the agent's mistaken assumption. It may mock away the provider behavior, assert only an HTTP status, or accept a patch that removes the desired side effect along with the bug.

Consider a webhook that grants seats twice when Paddle redelivers one transaction.completed event. A weak verifier asks:

Did the seat count stop increasing?

An agent can satisfy that check by granting no seats at all. The duplicate is gone, the customer receives nothing, and CI is green.

The real invariant is:

One purchase grants the purchased seats exactly once,
however many times the same event is delivered.

That distinction matters because CI is an optimization target. If “make the reported symptom disappear” is enough to pass, a plausible but destructive patch can pass.

An independent verifier must be able to reject the first fix.

That gate behavior showed up in a separate measured FetchSandbox acceptance run against a Paddle billing demo. The bug in that run was a webhook trusting a client-controlled seat quantity. The first proposed patch came back fix_incomplete; the agent iterated, and only the second patch produced:

buggy tree   → invariant VIOLATED
fixed tree   → invariant HELD
verdict      → green_allowed: true
proof grade  → measured

Inspect the measured client-controlled-seat receipt.

What does FetchSandbox verify on an AI-generated fix?

FetchSandbox acts as the integration verification layer between the coding agent and the pull request. Its prove_fix MCP tool runs the customer's real code before and after the proposed patch against the same provider failure scenario.

The sequence is:

The agent identifies an integration bug with find_bugs.
It proposes a diff with fix_bug.
Before writing that diff into the working tree, it calls prove_fix.
FetchSandbox materializes the still-broken project.
It applies the diff to a separate copy.
It runs the same provider scenario against both versions.
It returns green only when the bug reproduces before and the required invariant holds after.

If FetchSandbox cannot reproduce the bug, it does not convert uncertainty into a pass. The result is unproven, and green_allowed remains false.

A successful result has a machine-readable shape:

{
  "green_allowed": true,
  "state": "proven",
  "reproduced": true,
  "verified": true,
  "receipt_url": "https://fetchsandbox.com/runs/fix-...?flow=..."
}

The receipt URL is public to anyone who has the link. A reviewer can inspect it without access to the original agent conversation.

What should go into the pull request?

An AI-authored pull request should include the claim, the invariant, and the evidence. A short verification block is enough:

## Integration verification

- Failure: duplicate Paddle transaction webhook
- Invariant: one purchase grants seats exactly once
- Before patch: reproduced
- After patch: held
- Receipt: https://fetchsandbox.com/runs/fix-...?flow=...
- Remaining review: authorization, data migration, rollout safety

The receipt does not replace code review. It narrows code review to what still needs human judgment.

The reviewer can now check architecture, maintainability, security boundaries, and whether the invariant represents the intended business rule. They do not have to infer whether the agent ever ran the failure it claims to have fixed.

How does the proof continue into CI/CD?

The pre-merge receipt and the CI workflow gate answer different questions. The receipt proves that one proposed diff changed a reproduced failure into the required behavior. CI keeps the provider workflow from regressing as the branch changes.

FetchSandbox's CLI produces structured JSON and exits non-zero when a normal workflow run fails:

- name: Run API integration workflows
  run: |
    npx fetchsandbox run "$FETCHSANDBOX_ID" --all --json \
      > fetchsandbox-workflows.json

Store fetchsandbox-workflows.json as a build artifact and make the job a required status check when your team is comfortable with its coverage. The same workflow definitions can be explored locally through FetchSandbox MCP and run headlessly in the pipeline.

Today, the prove_fix receipt is attached to the pull request as a link. An automatic GitHub Action that posts the receipt and check result is not yet a shipped FetchSandbox feature, so do not describe that automation as live. The available path is explicit and auditable: prove through MCP, paste the receipt into the PR, and keep the provider workflows running in CI.

What does a proof receipt prove?

A FetchSandbox proof receipt proves the behavior shown on the artifact: the named scenario ran, the bug reproduced on the pre-patch code, and the measured invariant held or did not hold after the patch.

It does not prove that every requirement in the pull request is correct. It does not prove an untested security property, a safe database migration, or a production rollout plan. Good evidence has boundaries.

That is also why an unproven verdict is useful. “We could not reproduce this failure” tells the reviewer where the evidence ends. A confident green with no execution would be worse.

What changes when every pull request carries evidence?

The unit of review changes from a diff plus a confident summary to a diff plus a falsifiable result.

The coding agent can still generate quickly. The human still owns intent and risk. The verification layer owns one narrower job: make the provider failure happen, run the proposed fix against it, and preserve what happened in a receipt.

Cheap code is useful only when teams can qualify it for production without creating an equally large review queue. The next improvement in agentic software development is not another faster generator. It is a better way to say no to an unproven patch.

Connect FetchSandbox MCP to Cursor or Claude Code, run find_bugs → fix_bug → prove_fix, and attach the returned receipt to the next integration pull request. For broader workflow setup, see agent API workflow testing and API integration testing in CI.

Questions developers ask

Does passing CI prove AI-generated code is correct?

No. Passing CI proves only that the checks configured in CI passed. Verify that those checks exercise the relevant provider failure and assert the intended business state, not just an HTTP status or the absence of an exception.

Should the coding agent write its own tests?

The agent can write tests, but those tests should not be the only judge of its patch. Use independent scenarios, reviewed invariants, negative controls, or a separate verification system that can reject the agent's first answer.

Does FetchSandbox replace human pull-request review?

No. FetchSandbox supplies runtime evidence for API integration behavior. Humans still review intent, architecture, security, migrations, and rollout risk.

Can FetchSandbox automatically comment on every GitHub pull request?

Not currently. The shipped workflow returns a shareable proof receipt through MCP, which you can attach manually. The CLI can run workflows as a CI status check; automatic PR comments are planned rather than live.

What happens when FetchSandbox cannot reproduce the bug?

The proof declines to green. An unavailable or inconclusive scenario remains unproven instead of being reported as verified.

Test ElevenLabs 429 Handling Without Burning API Quota

FetchSandbox — Mon, 07 Sep 2026 14:21:39 +0000

Your app loads the ElevenLabs voice list successfully. The user picks a voice, clicks Generate, and the text-to-speech request returns 429.

That is a more useful test than an API that always rate-limits. Authentication worked. Discovery worked. The failure happened on the expensive operation, after the UI had already promised progress.

The branch most mocks skip

A generated ElevenLabs integration usually proves this sequence:

GET  /v1/voices                    -> 200
POST /v1/text-to-speech/{voiceId}  -> 200

The fixture is green forever. It does not answer:

What happens when voice discovery succeeds but generation is throttled?
Does the client retry immediately?
Does the SDK already retry before your wrapper retries?
Does the UI create several concurrent generation jobs?
Is there a maximum attempt count?
What happens when no Retry-After header is present?

Those are client-policy questions. Burning production quota until ElevenLabs returns 429 is an expensive way to discover the answers.

A 429 is not permission to retry forever

Rate-limit handling needs a decision function, not a recursive catch:

type RetryDecision =
  | { action: "wait"; delayMs: number }
  | { action: "stop"; reason: string };

function decide429(
  response: Response,
  attempt: number,
  remainingMs: number
): RetryDecision {
  if (attempt >= 3) {
    return { action: "stop", reason: "retry budget exhausted" };
  }

  const retryAfter = response.headers.get("retry-after");
  const fallbackMs = Math.min(1000 * 2 ** attempt, 8000);
  const delayMs = retryAfter
    ? parseRetryAfter(retryAfter)
    : fallbackMs;

  if (delayMs > remainingMs) {
    return { action: "stop", reason: "deadline exceeded" };
  }

  return { action: "wait", delayMs };
}

The important assertions are:

generation_attempts <= 3
voice_list_calls == 1
no duplicate UI jobs
fallback backoff used when Retry-After is absent
controlled error returned when the retry budget ends

Do not assert only that the client eventually threw. A tight loop can throw after hammering the provider twenty times.

Watch for stacked retries

Many SDKs retry transient failures internally. If your application wrapper also retries, the budgets multiply:

SDK attempts:          3
application attempts:  3
actual HTTP requests:  9

That is why the test needs a request count. Pick one layer to own the policy, or coordinate the limits explicitly.

The failure can get worse in a browser. A disabled button that re-enables too early allows the user to start another retry tree while the first is waiting.

Force the real workflow shape without an API key

FetchSandbox ships an ElevenLabs twin with a curated text_to_speech workflow:

GET /v1/voices to discover available voices
Select the returned voice_id
POST /v1/text-to-speech/{voice_id}

Under the rate_limited scenario, only the textToSpeech operation changes:

rate_limited:
  overrides:
    - operation_id: textToSpeech
      response_status: 429
      error_code: rate_limit_exceeded
      error_detail: "Rate limit exceeded. Please try again later."

The first step still succeeds. The generation step returns the provider-shaped 429. The configured response intentionally has no Retry-After, so this scenario tests your documented fallback backoff and retry ceiling rather than server-directed waiting.

No ElevenLabs API key and no quota consumption are involved.

With FetchSandbox MCP in Cursor or Claude:

Run elevenlabs text_to_speech under rate_limited.
Verify voice discovery succeeds and generation returns 429.
Count every generation attempt, prove the voice list is not fetched
again, and confirm the client stops at its retry budget.
Give me the run receipt.

The receipt provides the request sequence and exact failure. Your app-level test supplies the clock and UI assertions.

Make backoff deterministic

Do not make CI literally sleep through production delays. Inject time:

type RetryClock = {
  now(): number;
  sleep(ms: number): Promise<void>;
  random(): number;
};

Production uses the real clock. Tests use a fake clock and record requested delays:

attempt 1 -> 429 -> requested sleep 1000ms
attempt 2 -> 429 -> requested sleep 2000ms
attempt 3 -> 429 -> stop

You can prove exponential backoff, jitter bounds, cancellation, and the total deadline without adding seconds to every CI run.

Local first, pipeline next

Ask the coding agent to run the twin while it writes the policy. Then keep the same scenario in CI:

fetchsandbox run <sandbox-id> text_to_speech \
  --scenario rate_limited \
  --json

The local receipt is useful during review. The pipeline prevents a future refactor from changing maxAttempts = 3 into unbounded recursion.

Questions people ask

How can I test ElevenLabs rate limits without using quota?

Point the integration at a service twin and force textToSpeech to return 429. This exercises the client contract without sending audio-generation requests to ElevenLabs.

Should I always use `Retry-After`?

Honor a valid Retry-After when the provider sends one, subject to your own deadline and attempt cap. When it is absent, use a documented capped backoff with jitter. Test both branches separately.

Should a 429 restart the whole workflow?

No. Resume from the safe failed operation. In this workflow, do not fetch voices again or create a second local generation job merely because text-to-speech was throttled.

Is rate limiting the same as exhausted quota?

No. A temporary rate limit is generally pause-and-retry. A hard quota or billing limit is stop-and-alert. OpenAI, for example, can use HTTP 429 for both while distinguishing them with error codes such as rate_limit_exceeded and insufficient_quota.

Does FetchSandbox replace ElevenLabs testing?

No. Use the twin for repeatable negative paths and CI. Use ElevenLabs afterward for final credentials, model availability, audio quality, and real account limits.

The evidence to attach

Require a receipt showing one successful voice lookup followed by the forced 429, plus an app assertion showing the exact retry count and requested delays.

The canonical guide covers mid-workflow 429 testing without partner quota. Connect the twins through FetchSandbox MCP.

Your PayPal Refund Worked. Your Lovable App Says It Failed.

FetchSandbox — Mon, 07 Sep 2026 13:26:57 +0000

Your app called PayPal's refund endpoint. PayPal completed the operation, but the 201 Created response never reached your server.

The app retries and receives 422 CAPTURE_FULLY_REFUNDED. If the handler treats every non-2xx response as a failed refund, your customer and support team now see opposite versions of reality.

This guide shows the production-safe behavior and how to test it before using a real payment.

What should a PayPal refund handler do after a timeout?

Reuse the same PayPal-Request-Id, then reconcile PayPal's state before marking the refund failed.

PayPal uses PayPal-Request-Id as the idempotency key for REST POST operations. One refund operation should have one stored request ID:

type RefundRecord = {
  id: string;
  captureId: string;
  paypalRequestId: string;
  status: "pending" | "completed" | "reconcile_required" | "failed";
};

Send that ID on the first attempt:

const result = await fetch(
  `${PAYPAL_API}/v2/payments/captures/${refund.captureId}/refund`,
  {
    method: "POST",
    headers: {
      Authorization: `Bearer ${accessToken}`,
      "Content-Type": "application/json",
      "PayPal-Request-Id": refund.paypalRequestId,
    },
    body: JSON.stringify({
      amount: {
        currency_code: "USD",
        value: "25.00",
      },
    }),
  }
);

If the request times out, send the same request ID again. Do not create a new UUID per attempt.

Why does PayPal return `CAPTURE_FULLY_REFUNDED`?

It means the capture's refundable balance is already zero. Possible causes include:

the first request worked but its response was lost
another support tool issued the refund
the PayPal dashboard issued it
several partial refunds already reached the captured total

The response shape is carried inside a broader 422:

{
  "name": "UNPROCESSABLE_ENTITY",
  "details": [
    {
      "issue": "CAPTURE_FULLY_REFUNDED",
      "description": "The capture has already been fully refunded."
    }
  ]
}

Do not branch on status code alone. 422 can also mean the amount exceeds the remaining balance, a chargeback blocks the refund, or the account cannot perform the operation.

Parse details[].issue, then choose the business response.

The correct state transition

When the issue is CAPTURE_FULLY_REFUNDED, stop automatic retries and move the local operation to reconcile_required.

if (response.status === 422) {
  const body = await response.json();
  const issues = body.details?.map(
    (detail: { issue: string }) => detail.issue
  ) ?? [];

  if (issues.includes("CAPTURE_FULLY_REFUNDED")) {
    await markReconcileRequired(refund.id);
    await reconcilePayPalRefund(refund);
    return;
  }
}

Reconciliation should confirm the external refund ID, amount, currency, and status before completing the local row. Avoid changing CAPTURE_FULLY_REFUNDED directly into success because PayPal may have been updated by a different system.

The invariant is:

PayPal has refunded the intended amount
AND the local operation points to that external state
AND another retry cannot issue or record the refund again

How do I test this in a Lovable or Bolt app?

Use a stateful PayPal service twin to force the failure before deployment.

FetchSandbox includes a curated PayPal Payments workflow called capture_and_refund. Its normal path:

Reads capture 2GG279541U471931P
Requests a partial refund of $25.00 USD
Expects 201 Created
Emits PAYMENT.CAPTURE.REFUNDED

The refund_not_allowed scenario changes the second step to:

response_status: 422
error_code: CAPTURE_FULLY_REFUNDED
error_detail: "The capture has already been fully refunded."

Add FetchSandbox MCP to Cursor, Claude, or another MCP-capable agent and ask:

Run paypal-payments capture_and_refund with refund_not_allowed.
Inspect my handler for the exact 422 issue. Prove it stops retries
and reconciles before reporting a failure. Return the receipt URL.

The run receipt records the request sequence and exact error contract. Your app-level assertion checks the database and UI.

For CI:

fetchsandbox run <sandbox-id> capture_and_refund \
  --scenario refund_not_allowed \
  --json

This gives the generated integration a failure-path acceptance test instead of relying on a code reviewer to infer behavior from the happy path.

Can Bolt preview receive PayPal webhooks?

Not directly. PayPal sends webhooks to a publicly reachable HTTPS endpoint, while Bolt's browser-based preview is not a public webhook destination. Deploy to Bolt Cloud, Netlify, or another host for final PayPal webhook connectivity testing.

The service twin handles a different job: it lets the agent exercise provider state, API errors, and expected webhook events before that deployment. Use both layers:

FetchSandbox for repeatable failure branches during development
PayPal sandbox for final credentials and webhook connectivity
Production only after both paths pass

Does a service twin replace an API mock?

No. A mock is useful when a component needs a shaped response. A twin is useful when correctness depends on what happened in earlier calls.

Refund testing needs state:

captured amount
- previous partial refunds
= remaining refundable amount

That is why CAPTURE_FULLY_REFUNDED is a lifecycle test rather than a fixture test.

Pull-request checklist

Before merging an AI-generated PayPal refund flow, attach evidence that:

one refund operation stores one PayPal-Request-Id
timeout retries reuse that ID
the handler parses details[].issue
CAPTURE_FULLY_REFUNDED stops automatic retries
local state reconciles before the UI reports success or failure
the pipeline can reproduce the same 422 path

The complete canonical guide is Test PayPal Refunds in Lovable Before Production. For the wider workflow, see integration testing for AI-built apps.

Your AI builder shipped the Stripe code in an afternoon. Verifying it took a week.

FetchSandbox — Sun, 06 Sep 2026 04:20:21 +0000

A friend shipped a booking app last month. Lovable built the whole thing, Stripe Checkout included, in something like an afternoon. It looked finished. It was finished, in the sense that you could pay and get a confirmation email.

Then a customer got charged twice for one booking.

Not a double-click. One payment, one payment_intent.succeeded, delivered twice, because Stripe retries. The handler ran twice and created two bookings. Nobody caught it in review, and here is the uncomfortable part: the code was fine on the happy path, and nobody reviewing it had written a line of it.

That is the actual shape of the problem right now. Build time collapsed. Verification time did not.

The generated code is not the problem

This is where I expected to find sloppy AI output, and I didn't. The handler read exactly like something a competent developer would write on a first pass:

app.post('/webhook', async (req, res) => {
  const event = stripe.webhooks.constructEvent(req.body, sig, secret);

  if (event.type === 'payment_intent.succeeded') {
    await createBooking(event.data.object.metadata.booking_id);
    await sendConfirmation(event.data.object.receipt_email);
  }

  res.json({ received: true });
});

Signature verified. Correct event type. Returns 200. It will pass code review and it will pass a smoke test, because the first delivery does exactly what it should.

There is no idempotency key anywhere in it. The agent was never asked "what happens on the second delivery," so it never answered that question. Neither did the reviewer, because you cannot see a retry in a diff.

Why the usual tools miss it

Mocks return the response you configured. A mock does not decide to retry you. That is the whole point of a mock, and it is also why it cannot catch this class of bug.

Staging proves the happy path against real credentials. To see a duplicate delivery you have to wait for Stripe to genuinely retry, which means causing a timeout on purpose and waiting out a backoff. Nobody does this on a Tuesday.

stripe trigger fires a fresh, well-formed event. Useful, but every trigger is a first delivery. The bug lives in the second one.

So the failure mode is invisible to all three, and it surfaces when real money moves.

Force the second delivery

What I actually wanted was a Stripe that behaves like Stripe, including the annoying parts, on demand. That is what a service twin is: stateful, so a created PaymentIntent can be read back later, and scriptable, so you can ask for the retry instead of waiting for it.

FetchSandbox runs as an MCP server, so the agent that wrote the integration can drive it. In Cursor or Claude:

Run the Stripe accept_payment workflow, then re-run it under the
webhook_retries scenario. Tell me how many bookings exist afterwards
and give me the receipt URL.

webhook_retries delivers each webhook once, then replays it twice with the same upstream event id at backoff intervals, which is the behaviour that matters. A handler deduping on a per-delivery header still fails here, because that header changes on every retry while event.id does not.

The run came back with three bookings for one payment. Not an opinion about the code, a count.

The fix is the boring one:

if (await alreadyProcessed(event.id)) {
  return res.json({ received: true });
}
await recordProcessed(event.id);

Then re-run the same scenario and the count stays at one. That buggy-to-fixed flip is the thing worth having, because "the agent says it fixed it" and "the retry no longer double-books" are different claims.

The part that makes it reviewable

Every run produces a receipt at a shareable URL: the requests in order, the state the twin held, the webhook events including the replays, and a verdict. The verdict is separate from the HTTP status on purpose, so a run can return 200 at every step and still come back unproven.

That distinction is what makes this useful when you didn't write the code. You are no longer reviewing an unfamiliar diff and hoping. You are reading evidence that a specific failure was forced and handled. Put the URL in the PR.

The same run works from the CLI when you want the pipeline to enforce it rather than a person remembering:

fetchsandbox run <sandbox-id> accept_payment \
  --scenario webhook_retries --json

Local proof while building, the same check gating merges. The failure branch becomes a required step instead of a thing you meant to test.

Beyond Stripe

Payments are where it bites hardest, but the pattern is identical anywhere an integration has a lifecycle. Paddle, Resend, Twilio, Clerk, Descope, and AgentMail have curated workflows and failure scenarios; the wider catalogue covers 50+ APIs generated from their OpenAPI specs. Same loop each time: run the flow, force the failure, keep the receipt.

Setup is one MCP config block, and there is more on the approach for AI-built apps if you want the longer version.

The actual question

I keep meeting people whose apps went from idea to paying customers in a weekend, and whose integration testing story is "staging looked fine."

So, genuinely: what are you doing here? Is anyone verifying the retry, the dropped webhook, and the mid-flow auth failure before production, or is the honest answer that you find out from a support ticket? I am more interested in what people actually do than what we all agree we should do.

undefined === undefined — the auth bypass your AI wrote into your checkout

FetchSandbox — Thu, 13 Aug 2026 17:40:08 +0000

TL;DR

An auth check that uses === where both sides can be undefined is fail-open — a missing config turns it from locked to open.

The one secret missing from .env.example was the internal auth token. Unset in a deploy, undefined === undefined let every anonymous request through.

Default to deny: validate types, reject empty, compare in constant time, and write the test that sends no credentials with the env var unset.

The coding agent handed me an auth middleware. It passed review. I almost shipped it.

function protect(req, res, next) {
  if (req.headers['x-internal-token'] === process.env.INTERNAL_API_TOKEN) return next();
  return requireAuth()(req, res, next);
}

Four lines, and it reads fine. An internal service calls with a shared token and skips the login flow; everyone else falls through to real auth. Then I checked what happens when the token isn't set.

What the check actually did

INTERNAL_API_TOKEN was the one secret missing from .env.example. Every other key was there — Stripe, Clerk, Paddle — but not this one. So on a deploy where nobody thought to set it, process.env.INTERNAL_API_TOKEN is undefined.

Now a normal browser request comes in. It doesn't send an x-internal-token header, so req.headers['x-internal-token'] is also undefined.

undefined === undefined → true → return next().

Auth bypassed. Every anonymous request to /checkout is treated as a trusted internal call. The endpoint is safest when the token is set and wide open when it's missing — which is exactly backwards. A misconfigured auth check should get more restrictive, not less.

And there's a quieter problem even when the token is set: === on a secret isn't constant-time. It short-circuits on the first mismatched byte, so response timing leaks the token one character at a time.

The real check

Fail closed:

const crypto = require('crypto');

function hasInternalToken(req) {
  const expected = process.env.INTERNAL_API_TOKEN;
  if (typeof expected !== 'string' || expected.length === 0) return false; // no token configured → deny
  const provided = req.headers['x-internal-token'];
  if (typeof provided !== 'string' || provided.length === 0) return false;
  const a = Buffer.from(provided), b = Buffer.from(expected);
  if (a.length !== b.length) return false;
  return crypto.timingSafeEqual(a, b);
}

function protect(req, res, next) {
  if (hasInternalToken(req)) return next();
  return requireAuth()(req, res, next);
}

No configured token means no bypass. An empty or missing header means no bypass. The compare is constant-time, and length is checked first (timingSafeEqual throws on length mismatch). Then add INTERNAL_API_TOKEN to .env.example so it can never be silently omitted again.

Why agents write it this way

The happy path looks correct. Token set, caller sends the right one, x === y is true — review sees a plausible auth check and moves on. The model optimizes for "works when configured," and === fails open by default. It never had to reason about the unset case, because the unset case isn't in the example the code was written against.

Humans do this too — the fail-open default, the === on a secret, the config that's "obviously" always set. Agents just do it faster and ship it greener.

Fail closed, and prove it

The tell isn't in the code. It's in the test you didn't write. Send a request with no credentials and the env var unset, and assert it's rejected:

delete process.env.INTERNAL_API_TOKEN;
const res = await request(app).post('/checkout').send({ seats: 5 });
expect(res.status).not.toBe(200); // must be 401/302 — not "welcome in"

If an unauthenticated request with no token configured gets a 200, your auth is open — and no amount of green on the happy path will tell you. The bug lives entirely in the case the tests never exercise.

What I changed in the habit

Auth defaults to deny. Every branch that grants access starts from "no."
Validate both sides — type is string, length > 0. undefined === undefined is not authentication.
Constant-time compare for any secret (crypto.timingSafeEqual), never ===.
Write the misconfigured-env test — unset the token, send nothing, assert rejected.
Every secret the code reads goes in .env.example. A missing one isn't a config gap; here it was the whole lock.

I found this the way I find most of them: reproduced it on the real code — an anonymous request returned 200 — applied the fix, and confirmed it flipped to a 302, with FetchSandbox. The bug was never in the happy path. It was in the case the code never had to handle.

Which is the rule every auth check should start from: fail closed.

Green passed. The fix granted zero seats.

FetchSandbox — Tue, 11 Aug 2026 14:05:17 +0000

TL;DR

"Did the bug stop?" is a weak invariant. An over-suppressing fix — grant nothing — also stops the bug.

Assert the exact correct end state: after N identical deliveries, seats == the purchased amount. Not more. Not zero.

A verifier that cannot reject a deliberately broken fix is not a verifier. It is a green light.

The coding agent handed me a patch. CI went green. I almost merged it.

The original bug was ugly and familiar. A Stripe webhook handler granted seats on invoice.paid (or whatever your equivalent is — checkout.session.completed, customer.subscription.updated, pick your poison). It was not idempotent. Stripe retried. The handler granted again.

A five-seat purchase became 5, then 10, then 15. Same event. Same customer. Three deliveries.

That is the bug everyone warns you about. At-least-once delivery. Dedupe on event id. INSERT … ON CONFLICT DO NOTHING. You have heard the sermon.

So I asked the agent to fix it. It wrote a patch. My verifier ran the retry series and graded the result green.

Then I looked at the seats.

delivery     buggy handler     agent "fix"      what we wanted
1            5                 0                5
2            10                0                5
3            15                0                5

The count had stopped growing. True. It had also stopped granting.

Customers who paid got nothing. Including on the first, legitimate delivery. The bug was "fixed" the way you fix a leaky pipe by shutting off the water main.

What the check actually asserted

The verifier was not stupid in a cartoon way. It did replay the webhook. It did look at a side effect. It did not just grep the diff for idempotent.

It asserted the thing I had complained about: the count should not keep climbing.

Something like this:

// Weak: "the bug stopped"
const series = seatsAfterEachDelivery(handler); // e.g. [0, 0, 0]

expect(series[2]).toBeLessThanOrEqual(series[0]);
expect(new Set(series).size).toBe(1); // "stable across retries"

[5, 10, 15] fails. Good.

[5, 5, 5] passes. Also good.

[0, 0, 0] passes. That is the hole.

Any monotonicity check, any "didn't grow," any "delta is zero after the first call" that never pins the value will accept the over-suppressing patch. Delete the grant. Swallow the event. Return 200 and write nothing. The graph is flat. Green.

If you have ever written expect(errors).toHaveLength(0) and then watched someone delete the code that could error, you have met this family.

The real invariant

Idempotency is not "nothing happens twice." It is "the same request leaves the system in the same correct state."

For this handler, after N identical deliveries of a five-seat purchase, seats must equal exactly five. Not fifteen. Not zero.

// Exact: after N identical deliveries, seats == purchased amount
const series = seatsAfterEachDelivery(handler);

expect(series).toEqual([5, 5, 5]);

That one line rejects [5, 10, 15] and [0, 0, 0]. It accepts [5, 5, 5].

When I switched the assert, the zero-grant patch went red. The agent tried again. The patch that actually deduped — grant once, ignore the retries — produced [5, 5, 5] and stayed green.

Same deliveries. Same metric. Different question.

"Did it stop doing the bad thing?" vs "Is the world in the state a correct implementation would leave it in?"

Why agents fail this way

I do not think the model was being clever. I think it was being literal.

You said: stop double-granting. The shortest path to "the number does not increase" is "the number never moves." Skip the write. Make the insert always look like a conflict. Short-circuit before grantSeats. Plenty of ways to get a flat series.

Humans do this too under time pressure. Feature flags that default off. if (false) around the dangerous block. Tests that mock the collaborator into a no-op. Agents just do it faster, and they will happily stop at the first green.

Your CI is an optimization target. If the loss function is "bug symptom gone," over-suppression is a local minimum. If the loss function is "exact end state," that minimum disappears.

This is not a Stripe trivia item. It is any side-effecting handler you let an agent touch:

A refund webhook that "fixes" double-refunds by refunding $0.
A provisioner that "fixes" duplicate users by creating none.
A retry queue that "fixes" duplicate emails by sending nothing, including the first time.

The shape is always the same. Symptom: too much. Naive fix: zero. Correct fix: once.

Positive control, negative control

Here is the part I now require before I trust a verifier — mine, yours, an agent's, a CI job.

Positive control. A known-good implementation must pass. For me that was the idempotent grant: [5, 5, 5]. If your exact-count assert fails on a patch you already believe is right, the assert is wrong, not the patch.

Negative control. A known-bad implementation must fail. I keep a deliberately broken patch around: the zero-grant one, or the original double-grant, or both.

expect(seatsAfterEachDelivery(buggy)).toEqual([5, 5, 5]);
// fail: [5, 10, 15]

expect(seatsAfterEachDelivery(zeroGrant)).toEqual([5, 5, 5]);
// fail: [0, 0, 0]   ← this is the control most suites skip

expect(seatsAfterEachDelivery(idempotent)).toEqual([5, 5, 5]);
// pass

If the zero-grant still greens, you do not have a proof. You have a check that the symptom you first noticed got quieter.

I used to stop at the positive control. "The good fix passes, ship it." That is how [0, 0, 0] got a green. The suite had never been shown a lying patch.

This is the same idea as a mutation test, just less academic. You do not need a framework. You need one broken cousin of the fix, and a gate that refuses to call itself a verifier until that cousin fails.

What I changed in the habit

I still replay the webhook three times. That part was never the mistake. The mistake was scoring the replay with a bound instead of an equality.

The checklist I use now, when an agent "fixes" a production-shaped bug:

Write the end state in English first. "After three identical invoice.paid deliveries for a 5-seat price, seat_limit == 5."
Put that number in the assert. Not <=. Not "unchanged after first." The number.
Run the original bug. It must fail that assert.
Run a spiteful fix (no-op, always-conflict, grant-zero). It must fail too.
Then run the agent's patch.

Step 4 is the one I had skipped. It is also the cheapest. You can write the no-op in thirty seconds. If your suite cannot fail it, do not let the suite pass the agent.

I hit this while building FetchSandbox, replaying Stripe deliveries against persistent seat state instead of a one-shot fixture.

The transferable rule is not about Stripe, and it is not about my tooling. When you verify an AI-written fix, "did the bug stop?" is the wrong question. Ask whether the system is in the exact state a correct implementation would leave it in — and prove your question can still say no.

DEV Community: FetchSandbox

Lovable Builders Break One Action On Purpose. Nobody Named It.

What breaks after the happy path

Break the action that can fail quietly

Replit's Green Twin Run Never Hit Its App

Make the app part of the test

Lovable's MCP Form Has a None Option. Don't Pick It.

Get the key first

Then the form

Then make it misbehave

Deterministic Evals for Coding Agents: Prove the Fix

What is a deterministic eval engine for coding agents?

Why is an LLM judge not enough to verify a code fix?

How does the exit-code flip decide whether a fix is proven?

Why must the same failure run on old and fixed code?

How does FetchSandbox inject integration failures?

Is there an LLM anywhere in the FetchSandbox eval path?

What does a coding agent send to prove_fix?

What is in the proof receipt?

How does the proof fit into PR review and CI/CD?

Does FetchSandbox evaluate every kind of coding task?

How do I use FetchSandbox from a coding agent?

Questions about deterministic coding-agent evals

What is the difference between a coding-agent benchmark and a fix proof?

Can the same coding agent generate the fix and request the proof?

Why not trust a test that the coding agent wrote?

What happens if the old code passes?

What happens if the runner or probe crashes?

Can an LLM-generated probe produce a green result?

RevenueCat Returned 200. The Wallet Still Went Backward.

What the handler assumed

The three clocks that disagree

The doorbell pattern

The test happy-path fixtures hide

Prove it from the coding agent, then keep it in CI

Your Paddle Webhook Is Signed. The Seat Count Is Not.

The two quantities inside one valid event

The handler that looked secure

What the fixed trust boundary looks like

The test that exposes the bug

Run it from the coding agent

Carry the proof into review and CI

Signed does not mean trusted

AI Code Is Cheap. Your Pull Request Still Needs Proof.

How should you verify AI-generated code before merging it?

Why is reviewing AI-written code becoming the bottleneck?

Why is a green test written by the same agent not enough?

What does FetchSandbox verify on an AI-generated fix?

What should go into the pull request?

How does the proof continue into CI/CD?

What does a proof receipt prove?

What changes when every pull request carries evidence?

Questions developers ask

Does passing CI prove AI-generated code is correct?

Should the coding agent write its own tests?

Does FetchSandbox replace human pull-request review?

Can FetchSandbox automatically comment on every GitHub pull request?

What happens when FetchSandbox cannot reproduce the bug?

Test ElevenLabs 429 Handling Without Burning API Quota

The branch most mocks skip

A 429 is not permission to retry forever

Watch for stacked retries

Force the real workflow shape without an API key

Make backoff deterministic

Local first, pipeline next

Questions people ask

How can I test ElevenLabs rate limits without using quota?

Should I always use Retry-After?

Should a 429 restart the whole workflow?

Is rate limiting the same as exhausted quota?

Does FetchSandbox replace ElevenLabs testing?

The evidence to attach

Your PayPal Refund Worked. Your Lovable App Says It Failed.

What should a PayPal refund handler do after a timeout?

Why does PayPal return CAPTURE_FULLY_REFUNDED?

The correct state transition

How do I test this in a Lovable or Bolt app?

Can Bolt preview receive PayPal webhooks?

Does a service twin replace an API mock?

Pull-request checklist

What does a coding agent send to `prove_fix`?

Should I always use `Retry-After`?

Why does PayPal return `CAPTURE_FULLY_REFUNDED`?