<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: FetchSandbox</title>
    <description>The latest articles on DEV Community by FetchSandbox (@fetchsandbox).</description>
    <link>https://dev.to/fetchsandbox</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3830177%2Fe63c5a3c-3f48-4ae2-ad6b-ed510712a396.png</url>
      <title>DEV Community: FetchSandbox</title>
      <link>https://dev.to/fetchsandbox</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/fetchsandbox"/>
    <language>en</language>
    <item>
      <title>AI Code Is Cheap. Your Pull Request Still Needs Proof.</title>
      <dc:creator>FetchSandbox</dc:creator>
      <pubDate>Tue, 08 Sep 2026 20:59:59 +0000</pubDate>
      <link>https://dev.to/fetchsandbox/ai-code-is-cheap-your-pull-request-still-needs-proof-5hb2</link>
      <guid>https://dev.to/fetchsandbox/ai-code-is-cheap-your-pull-request-still-needs-proof-5hb2</guid>
      <description>&lt;p&gt;Your coding agent opens a plausible pull request before lunch. The tests it wrote pass, the diff reads cleanly, and the reviewer is left with the expensive question: &lt;strong&gt;did this change survive the provider failure it claims to handle?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Writing the integration is now the cheap step. Establishing that the integration behaves correctly under retries, lost responses, stale events, webhooks, and partial failure is still the hard step.&lt;/p&gt;

&lt;h2&gt;
  
  
  How should you verify AI-generated code before merging it?
&lt;/h2&gt;

&lt;p&gt;Verify AI-generated integration code with an independent runtime check, not only the agent's explanation or the tests it wrote. Reproduce the provider failure against the still-broken code, apply the proposed diff to a copy, run the same failure again, and require a measured broken-to-fixed result before merge.&lt;/p&gt;

&lt;p&gt;The output should be a shareable receipt that records:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;the failure scenario that ran&lt;/li&gt;
&lt;li&gt;whether the bug reproduced before the patch&lt;/li&gt;
&lt;li&gt;whether the invariant held after the patch&lt;/li&gt;
&lt;li&gt;the requests, responses, and observed side effects&lt;/li&gt;
&lt;li&gt;an explicit proven, rejected, or unproven verdict&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That receipt changes the pull-request conversation from “the agent says this works” to “here is what ran, and here is what changed.”&lt;/p&gt;

&lt;h2&gt;
  
  
  Why is reviewing AI-written code becoming the bottleneck?
&lt;/h2&gt;

&lt;p&gt;AI coding agents can produce more diffs than a team can carefully inspect. Faster generation increases review pressure because plausible code still has to be checked against the real behavior of every external system it touches.&lt;/p&gt;

&lt;p&gt;Static review remains useful. It can catch an unsafe query, a missing authorization check, or an obviously incorrect SDK call. But many integration bugs do not live on one suspicious line:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;the same webhook is delivered twice&lt;/li&gt;
&lt;li&gt;a successful response disappears after the provider commits the write&lt;/li&gt;
&lt;li&gt;a stale event arrives after a newer event&lt;/li&gt;
&lt;li&gt;the first request creates an ID that the second request never persists&lt;/li&gt;
&lt;li&gt;a handler returns &lt;code&gt;200&lt;/code&gt; but leaves local state unchanged&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The diff can look reasonable while the lifecycle is still broken.&lt;/p&gt;

&lt;p&gt;Most current writing about AI code verification says the same broad thing: define acceptance criteria, run tests, and keep humans in the loop. That advice is correct, but it usually stops before the reviewer gets an inspectable artifact showing the failure on the actual code.&lt;/p&gt;

&lt;p&gt;The missing layer is &lt;strong&gt;runtime evidence attached to the change&lt;/strong&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why is a green test written by the same agent not enough?
&lt;/h2&gt;

&lt;p&gt;A test written by the same agent can repeat the agent's mistaken assumption. It may mock away the provider behavior, assert only an HTTP status, or accept a patch that removes the desired side effect along with the bug.&lt;/p&gt;

&lt;p&gt;Consider a webhook that grants seats twice when Paddle redelivers one &lt;code&gt;transaction.completed&lt;/code&gt; event. A weak verifier asks:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Did the seat count stop increasing?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;An agent can satisfy that check by granting no seats at all. The duplicate is gone, the customer receives nothing, and CI is green.&lt;/p&gt;

&lt;p&gt;The real invariant is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;One purchase grants the purchased seats exactly once,
however many times the same event is delivered.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That distinction matters because CI is an optimization target. If “make the reported symptom disappear” is enough to pass, a plausible but destructive patch can pass.&lt;/p&gt;

&lt;p&gt;An independent verifier must be able to reject the first fix.&lt;/p&gt;

&lt;p&gt;That gate behavior showed up in a separate measured FetchSandbox acceptance run against a Paddle billing demo. The bug in that run was a webhook trusting a client-controlled seat quantity. The first proposed patch came back &lt;code&gt;fix_incomplete&lt;/code&gt;; the agent iterated, and only the second patch produced:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;buggy tree   → invariant VIOLATED
fixed tree   → invariant HELD
verdict      → green_allowed: true
proof grade  → measured
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://fetchsandbox.com/runs/fix-417c74290218?flow=9c61192f9179" rel="noopener noreferrer"&gt;Inspect the measured client-controlled-seat receipt&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  What does FetchSandbox verify on an AI-generated fix?
&lt;/h2&gt;

&lt;p&gt;FetchSandbox acts as the integration verification layer between the coding agent and the pull request. Its &lt;code&gt;prove_fix&lt;/code&gt; MCP tool runs the customer's real code before and after the proposed patch against the same provider failure scenario.&lt;/p&gt;

&lt;p&gt;The sequence is:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The agent identifies an integration bug with &lt;code&gt;find_bugs&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;It proposes a diff with &lt;code&gt;fix_bug&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Before writing that diff into the working tree, it calls &lt;code&gt;prove_fix&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;FetchSandbox materializes the still-broken project.&lt;/li&gt;
&lt;li&gt;It applies the diff to a separate copy.&lt;/li&gt;
&lt;li&gt;It runs the same provider scenario against both versions.&lt;/li&gt;
&lt;li&gt;It returns green only when the bug reproduces before and the required invariant holds after.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;If FetchSandbox cannot reproduce the bug, it does not convert uncertainty into a pass. The result is unproven, and &lt;code&gt;green_allowed&lt;/code&gt; remains false.&lt;/p&gt;

&lt;p&gt;A successful result has a machine-readable shape:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"green_allowed"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"state"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"proven"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"reproduced"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"verified"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"receipt_url"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"https://fetchsandbox.com/runs/fix-...?flow=..."&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The receipt URL is public to anyone who has the link. A reviewer can inspect it without access to the original agent conversation.&lt;/p&gt;

&lt;h2&gt;
  
  
  What should go into the pull request?
&lt;/h2&gt;

&lt;p&gt;An AI-authored pull request should include the claim, the invariant, and the evidence. A short verification block is enough:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight markdown"&gt;&lt;code&gt;&lt;span class="gu"&gt;## Integration verification&lt;/span&gt;
&lt;span class="p"&gt;
-&lt;/span&gt; Failure: duplicate Paddle transaction webhook
&lt;span class="p"&gt;-&lt;/span&gt; Invariant: one purchase grants seats exactly once
&lt;span class="p"&gt;-&lt;/span&gt; Before patch: reproduced
&lt;span class="p"&gt;-&lt;/span&gt; After patch: held
&lt;span class="p"&gt;-&lt;/span&gt; Receipt: https://fetchsandbox.com/runs/fix-...?flow=...
&lt;span class="p"&gt;-&lt;/span&gt; Remaining review: authorization, data migration, rollout safety
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The receipt does not replace code review. It narrows code review to what still needs human judgment.&lt;/p&gt;

&lt;p&gt;The reviewer can now check architecture, maintainability, security boundaries, and whether the invariant represents the intended business rule. They do not have to infer whether the agent ever ran the failure it claims to have fixed.&lt;/p&gt;

&lt;h2&gt;
  
  
  How does the proof continue into CI/CD?
&lt;/h2&gt;

&lt;p&gt;The pre-merge receipt and the CI workflow gate answer different questions. The receipt proves that one proposed diff changed a reproduced failure into the required behavior. CI keeps the provider workflow from regressing as the branch changes.&lt;/p&gt;

&lt;p&gt;FetchSandbox's CLI produces structured JSON and exits non-zero when a normal workflow run fails:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Run API integration workflows&lt;/span&gt;
  &lt;span class="na"&gt;run&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;|&lt;/span&gt;
    &lt;span class="s"&gt;npx fetchsandbox run "$FETCHSANDBOX_ID" --all --json \&lt;/span&gt;
      &lt;span class="s"&gt;&amp;gt; fetchsandbox-workflows.json&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Store &lt;code&gt;fetchsandbox-workflows.json&lt;/code&gt; as a build artifact and make the job a required status check when your team is comfortable with its coverage. The same workflow definitions can be explored locally through &lt;a href="https://dev.to/mcp"&gt;FetchSandbox MCP&lt;/a&gt; and run headlessly in the pipeline.&lt;/p&gt;

&lt;p&gt;Today, the &lt;code&gt;prove_fix&lt;/code&gt; receipt is attached to the pull request as a link. An automatic GitHub Action that posts the receipt and check result is not yet a shipped FetchSandbox feature, so do not describe that automation as live. The available path is explicit and auditable: prove through MCP, paste the receipt into the PR, and keep the provider workflows running in CI.&lt;/p&gt;

&lt;h2&gt;
  
  
  What does a proof receipt prove?
&lt;/h2&gt;

&lt;p&gt;A FetchSandbox proof receipt proves the behavior shown on the artifact: the named scenario ran, the bug reproduced on the pre-patch code, and the measured invariant held or did not hold after the patch.&lt;/p&gt;

&lt;p&gt;It does not prove that every requirement in the pull request is correct. It does not prove an untested security property, a safe database migration, or a production rollout plan. Good evidence has boundaries.&lt;/p&gt;

&lt;p&gt;That is also why an unproven verdict is useful. “We could not reproduce this failure” tells the reviewer where the evidence ends. A confident green with no execution would be worse.&lt;/p&gt;

&lt;h2&gt;
  
  
  What changes when every pull request carries evidence?
&lt;/h2&gt;

&lt;p&gt;The unit of review changes from a diff plus a confident summary to a diff plus a falsifiable result.&lt;/p&gt;

&lt;p&gt;The coding agent can still generate quickly. The human still owns intent and risk. The verification layer owns one narrower job: make the provider failure happen, run the proposed fix against it, and preserve what happened in a receipt.&lt;/p&gt;

&lt;p&gt;Cheap code is useful only when teams can qualify it for production without creating an equally large review queue. The next improvement in agentic software development is not another faster generator. It is a better way to say &lt;strong&gt;no&lt;/strong&gt; to an unproven patch.&lt;/p&gt;

&lt;p&gt;Connect &lt;a href="https://dev.to/mcp"&gt;FetchSandbox MCP&lt;/a&gt; to Cursor or Claude Code, run &lt;code&gt;find_bugs → fix_bug → prove_fix&lt;/code&gt;, and attach the returned receipt to the next integration pull request. For broader workflow setup, see &lt;a href="https://dev.to/agent-api-workflow-testing"&gt;agent API workflow testing&lt;/a&gt; and &lt;a href="https://dev.to/api-integration-testing-in-ci"&gt;API integration testing in CI&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Questions developers ask
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Does passing CI prove AI-generated code is correct?
&lt;/h3&gt;

&lt;p&gt;No. Passing CI proves only that the checks configured in CI passed. Verify that those checks exercise the relevant provider failure and assert the intended business state, not just an HTTP status or the absence of an exception.&lt;/p&gt;

&lt;h3&gt;
  
  
  Should the coding agent write its own tests?
&lt;/h3&gt;

&lt;p&gt;The agent can write tests, but those tests should not be the only judge of its patch. Use independent scenarios, reviewed invariants, negative controls, or a separate verification system that can reject the agent's first answer.&lt;/p&gt;

&lt;h3&gt;
  
  
  Does FetchSandbox replace human pull-request review?
&lt;/h3&gt;

&lt;p&gt;No. FetchSandbox supplies runtime evidence for API integration behavior. Humans still review intent, architecture, security, migrations, and rollout risk.&lt;/p&gt;

&lt;h3&gt;
  
  
  Can FetchSandbox automatically comment on every GitHub pull request?
&lt;/h3&gt;

&lt;p&gt;Not currently. The shipped workflow returns a shareable proof receipt through MCP, which you can attach manually. The CLI can run workflows as a CI status check; automatic PR comments are planned rather than live.&lt;/p&gt;

&lt;h3&gt;
  
  
  What happens when FetchSandbox cannot reproduce the bug?
&lt;/h3&gt;

&lt;p&gt;The proof declines to green. An unavailable or inconclusive scenario remains unproven instead of being reported as verified.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>devops</category>
      <category>ci</category>
      <category>mcp</category>
    </item>
    <item>
      <title>Test ElevenLabs 429 Handling Without Burning API Quota</title>
      <dc:creator>FetchSandbox</dc:creator>
      <pubDate>Mon, 07 Sep 2026 14:21:39 +0000</pubDate>
      <link>https://dev.to/fetchsandbox/test-elevenlabs-429-handling-without-burning-api-quota-o6m</link>
      <guid>https://dev.to/fetchsandbox/test-elevenlabs-429-handling-without-burning-api-quota-o6m</guid>
      <description>&lt;p&gt;Your app loads the ElevenLabs voice list successfully. The user picks a voice, clicks Generate, and the text-to-speech request returns &lt;code&gt;429&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;That is a more useful test than an API that always rate-limits. Authentication worked. Discovery worked. The failure happened on the expensive operation, after the UI had already promised progress.&lt;/p&gt;

&lt;h2&gt;
  
  
  The branch most mocks skip
&lt;/h2&gt;

&lt;p&gt;A generated ElevenLabs integration usually proves this sequence:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;GET  /v1/voices                    -&amp;gt; 200
POST /v1/text-to-speech/{voiceId}  -&amp;gt; 200
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The fixture is green forever. It does not answer:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;What happens when voice discovery succeeds but generation is throttled?&lt;/li&gt;
&lt;li&gt;Does the client retry immediately?&lt;/li&gt;
&lt;li&gt;Does the SDK already retry before your wrapper retries?&lt;/li&gt;
&lt;li&gt;Does the UI create several concurrent generation jobs?&lt;/li&gt;
&lt;li&gt;Is there a maximum attempt count?&lt;/li&gt;
&lt;li&gt;What happens when no &lt;code&gt;Retry-After&lt;/code&gt; header is present?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Those are client-policy questions. Burning production quota until ElevenLabs returns 429 is an expensive way to discover the answers.&lt;/p&gt;

&lt;h2&gt;
  
  
  A 429 is not permission to retry forever
&lt;/h2&gt;

&lt;p&gt;Rate-limit handling needs a decision function, not a recursive catch:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;type&lt;/span&gt; &lt;span class="nx"&gt;RetryDecision&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt;
  &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;action&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;wait&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nl"&gt;delayMs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;number&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;action&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;stop&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nl"&gt;reason&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt; &lt;span class="p"&gt;};&lt;/span&gt;

&lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;decide429&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="nx"&gt;response&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;Response&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nx"&gt;attempt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;number&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nx"&gt;remainingMs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;number&lt;/span&gt;
&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="nx"&gt;RetryDecision&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;attempt&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;action&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;stop&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;reason&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;retry budget exhausted&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="p"&gt;};&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;retryAfter&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;retry-after&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;fallbackMs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;Math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;min&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1000&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt; &lt;span class="nx"&gt;attempt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;8000&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;delayMs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;retryAfter&lt;/span&gt;
    &lt;span class="p"&gt;?&lt;/span&gt; &lt;span class="nf"&gt;parseRetryAfter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;retryAfter&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;fallbackMs&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;delayMs&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;remainingMs&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;action&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;stop&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;reason&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;deadline exceeded&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="p"&gt;};&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;action&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;wait&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;delayMs&lt;/span&gt; &lt;span class="p"&gt;};&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The important assertions are:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;generation_attempts &amp;lt;= 3
voice_list_calls == 1
no duplicate UI jobs
fallback backoff used when Retry-After is absent
controlled error returned when the retry budget ends
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Do not assert only that the client eventually threw. A tight loop can throw after hammering the provider twenty times.&lt;/p&gt;

&lt;h2&gt;
  
  
  Watch for stacked retries
&lt;/h2&gt;

&lt;p&gt;Many SDKs retry transient failures internally. If your application wrapper also retries, the budgets multiply:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;SDK attempts:          3
application attempts:  3
actual HTTP requests:  9
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That is why the test needs a request count. Pick one layer to own the policy, or coordinate the limits explicitly.&lt;/p&gt;

&lt;p&gt;The failure can get worse in a browser. A disabled button that re-enables too early allows the user to start another retry tree while the first is waiting.&lt;/p&gt;

&lt;h2&gt;
  
  
  Force the real workflow shape without an API key
&lt;/h2&gt;

&lt;p&gt;FetchSandbox ships an ElevenLabs twin with a curated &lt;code&gt;text_to_speech&lt;/code&gt; workflow:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;code&gt;GET /v1/voices&lt;/code&gt; to discover available voices&lt;/li&gt;
&lt;li&gt;Select the returned &lt;code&gt;voice_id&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;&lt;code&gt;POST /v1/text-to-speech/{voice_id}&lt;/code&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Under the &lt;code&gt;rate_limited&lt;/code&gt; scenario, only the &lt;code&gt;textToSpeech&lt;/code&gt; operation changes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;rate_limited&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;overrides&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;operation_id&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;textToSpeech&lt;/span&gt;
      &lt;span class="na"&gt;response_status&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;429&lt;/span&gt;
      &lt;span class="na"&gt;error_code&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;rate_limit_exceeded&lt;/span&gt;
      &lt;span class="na"&gt;error_detail&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Rate&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;limit&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;exceeded.&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;Please&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;try&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;again&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;later."&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The first step still succeeds. The generation step returns the provider-shaped 429. The configured response intentionally has no &lt;code&gt;Retry-After&lt;/code&gt;, so this scenario tests your documented fallback backoff and retry ceiling rather than server-directed waiting.&lt;/p&gt;

&lt;p&gt;No ElevenLabs API key and no quota consumption are involved.&lt;/p&gt;

&lt;p&gt;With &lt;a href="https://fetchsandbox.com/mcp" rel="noopener noreferrer"&gt;FetchSandbox MCP&lt;/a&gt; in Cursor or Claude:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Run elevenlabs text_to_speech under rate_limited.
Verify voice discovery succeeds and generation returns 429.
Count every generation attempt, prove the voice list is not fetched
again, and confirm the client stops at its retry budget.
Give me the run receipt.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The receipt provides the request sequence and exact failure. Your app-level test supplies the clock and UI assertions.&lt;/p&gt;

&lt;h2&gt;
  
  
  Make backoff deterministic
&lt;/h2&gt;

&lt;p&gt;Do not make CI literally sleep through production delays. Inject time:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;type&lt;/span&gt; &lt;span class="nx"&gt;RetryClock&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nf"&gt;now&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt; &lt;span class="kr"&gt;number&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="nf"&gt;sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="na"&gt;ms&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;number&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="nb"&gt;Promise&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="k"&gt;void&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="nf"&gt;random&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt; &lt;span class="kr"&gt;number&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;};&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Production uses the real clock. Tests use a fake clock and record requested delays:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;attempt 1 -&amp;gt; 429 -&amp;gt; requested sleep 1000ms
attempt 2 -&amp;gt; 429 -&amp;gt; requested sleep 2000ms
attempt 3 -&amp;gt; 429 -&amp;gt; stop
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You can prove exponential backoff, jitter bounds, cancellation, and the total deadline without adding seconds to every CI run.&lt;/p&gt;

&lt;h2&gt;
  
  
  Local first, pipeline next
&lt;/h2&gt;

&lt;p&gt;Ask the coding agent to run the twin while it writes the policy. Then keep the same scenario in CI:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;fetchsandbox run &amp;lt;sandbox-id&amp;gt; text_to_speech &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--scenario&lt;/span&gt; rate_limited &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--json&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The local receipt is useful during review. The pipeline prevents a future refactor from changing &lt;code&gt;maxAttempts = 3&lt;/code&gt; into unbounded recursion.&lt;/p&gt;

&lt;h2&gt;
  
  
  Questions people ask
&lt;/h2&gt;

&lt;h3&gt;
  
  
  How can I test ElevenLabs rate limits without using quota?
&lt;/h3&gt;

&lt;p&gt;Point the integration at a service twin and force &lt;code&gt;textToSpeech&lt;/code&gt; to return 429. This exercises the client contract without sending audio-generation requests to ElevenLabs.&lt;/p&gt;

&lt;h3&gt;
  
  
  Should I always use &lt;code&gt;Retry-After&lt;/code&gt;?
&lt;/h3&gt;

&lt;p&gt;Honor a valid &lt;code&gt;Retry-After&lt;/code&gt; when the provider sends one, subject to your own deadline and attempt cap. When it is absent, use a documented capped backoff with jitter. Test both branches separately.&lt;/p&gt;

&lt;h3&gt;
  
  
  Should a 429 restart the whole workflow?
&lt;/h3&gt;

&lt;p&gt;No. Resume from the safe failed operation. In this workflow, do not fetch voices again or create a second local generation job merely because text-to-speech was throttled.&lt;/p&gt;

&lt;h3&gt;
  
  
  Is rate limiting the same as exhausted quota?
&lt;/h3&gt;

&lt;p&gt;No. A temporary rate limit is generally pause-and-retry. A hard quota or billing limit is stop-and-alert. OpenAI, for example, can use HTTP 429 for both while distinguishing them with error codes such as &lt;code&gt;rate_limit_exceeded&lt;/code&gt; and &lt;code&gt;insufficient_quota&lt;/code&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  Does FetchSandbox replace ElevenLabs testing?
&lt;/h3&gt;

&lt;p&gt;No. Use the twin for repeatable negative paths and CI. Use ElevenLabs afterward for final credentials, model availability, audio quality, and real account limits.&lt;/p&gt;

&lt;h2&gt;
  
  
  The evidence to attach
&lt;/h2&gt;

&lt;p&gt;Require a receipt showing one successful voice lookup followed by the forced 429, plus an app assertion showing the exact retry count and requested delays.&lt;/p&gt;

&lt;p&gt;The canonical guide covers &lt;a href="https://fetchsandbox.com/blog/test-mid-workflow-429-without-burning-quota" rel="noopener noreferrer"&gt;mid-workflow 429 testing without partner quota&lt;/a&gt;. Connect the twins through &lt;a href="https://fetchsandbox.com/mcp" rel="noopener noreferrer"&gt;FetchSandbox MCP&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>elevenlabs</category>
      <category>ratelimit</category>
      <category>testing</category>
      <category>mcp</category>
    </item>
    <item>
      <title>Your PayPal Refund Worked. Your Lovable App Says It Failed.</title>
      <dc:creator>FetchSandbox</dc:creator>
      <pubDate>Mon, 07 Sep 2026 13:26:57 +0000</pubDate>
      <link>https://dev.to/fetchsandbox/your-paypal-refund-worked-your-lovable-app-says-it-failed-5gh2</link>
      <guid>https://dev.to/fetchsandbox/your-paypal-refund-worked-your-lovable-app-says-it-failed-5gh2</guid>
      <description>&lt;p&gt;Your app called PayPal's refund endpoint. PayPal completed the operation, but the &lt;code&gt;201 Created&lt;/code&gt; response never reached your server.&lt;/p&gt;

&lt;p&gt;The app retries and receives &lt;code&gt;422 CAPTURE_FULLY_REFUNDED&lt;/code&gt;. If the handler treats every non-2xx response as a failed refund, your customer and support team now see opposite versions of reality.&lt;/p&gt;

&lt;p&gt;This guide shows the production-safe behavior and how to test it before using a real payment.&lt;/p&gt;

&lt;h2&gt;
  
  
  What should a PayPal refund handler do after a timeout?
&lt;/h2&gt;

&lt;p&gt;Reuse the same &lt;code&gt;PayPal-Request-Id&lt;/code&gt;, then reconcile PayPal's state before marking the refund failed.&lt;/p&gt;

&lt;p&gt;PayPal uses &lt;code&gt;PayPal-Request-Id&lt;/code&gt; as the idempotency key for REST &lt;code&gt;POST&lt;/code&gt; operations. One refund operation should have one stored request ID:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;type&lt;/span&gt; &lt;span class="nx"&gt;RefundRecord&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="na"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="nl"&gt;captureId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="nl"&gt;paypalRequestId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="nl"&gt;status&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;pending&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;completed&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;reconcile_required&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;failed&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;};&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Send that ID on the first attempt:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;PAYPAL_API&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;/v2/payments/captures/&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;refund&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;captureId&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;/refund`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="na"&gt;method&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;POST&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="na"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="na"&gt;Authorization&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;`Bearer &lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;accessToken&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;Content-Type&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;application/json&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;PayPal-Request-Id&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;refund&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;paypalRequestId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;},&lt;/span&gt;
    &lt;span class="na"&gt;body&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;JSON&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;stringify&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
      &lt;span class="na"&gt;amount&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="na"&gt;currency_code&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;USD&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="na"&gt;value&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;25.00&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="p"&gt;},&lt;/span&gt;
    &lt;span class="p"&gt;}),&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If the request times out, send the same request ID again. Do not create a new UUID per attempt.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why does PayPal return &lt;code&gt;CAPTURE_FULLY_REFUNDED&lt;/code&gt;?
&lt;/h2&gt;

&lt;p&gt;It means the capture's refundable balance is already zero. Possible causes include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;the first request worked but its response was lost&lt;/li&gt;
&lt;li&gt;another support tool issued the refund&lt;/li&gt;
&lt;li&gt;the PayPal dashboard issued it&lt;/li&gt;
&lt;li&gt;several partial refunds already reached the captured total&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The response shape is carried inside a broader &lt;code&gt;422&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"UNPROCESSABLE_ENTITY"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"details"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"issue"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"CAPTURE_FULLY_REFUNDED"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"description"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"The capture has already been fully refunded."&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Do not branch on status code alone. &lt;code&gt;422&lt;/code&gt; can also mean the amount exceeds the remaining balance, a chargeback blocks the refund, or the account cannot perform the operation.&lt;/p&gt;

&lt;p&gt;Parse &lt;code&gt;details[].issue&lt;/code&gt;, then choose the business response.&lt;/p&gt;

&lt;h2&gt;
  
  
  The correct state transition
&lt;/h2&gt;

&lt;p&gt;When the issue is &lt;code&gt;CAPTURE_FULLY_REFUNDED&lt;/code&gt;, stop automatic retries and move the local operation to &lt;code&gt;reconcile_required&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;status&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="mi"&gt;422&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;body&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;issues&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;body&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;details&lt;/span&gt;&lt;span class="p"&gt;?.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;detail&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;issue&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;detail&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;issue&lt;/span&gt;
  &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="p"&gt;[];&lt;/span&gt;

  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;issues&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;includes&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;CAPTURE_FULLY_REFUNDED&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;markReconcileRequired&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;refund&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;reconcilePayPalRefund&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;refund&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Reconciliation should confirm the external refund ID, amount, currency, and status before completing the local row. Avoid changing &lt;code&gt;CAPTURE_FULLY_REFUNDED&lt;/code&gt; directly into success because PayPal may have been updated by a different system.&lt;/p&gt;

&lt;p&gt;The invariant is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;PayPal has refunded the intended amount
AND the local operation points to that external state
AND another retry cannot issue or record the refund again
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  How do I test this in a Lovable or Bolt app?
&lt;/h2&gt;

&lt;p&gt;Use a stateful PayPal service twin to force the failure before deployment.&lt;/p&gt;

&lt;p&gt;FetchSandbox includes a curated PayPal Payments workflow called &lt;code&gt;capture_and_refund&lt;/code&gt;. Its normal path:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Reads capture &lt;code&gt;2GG279541U471931P&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;Requests a partial refund of &lt;code&gt;$25.00 USD&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;Expects &lt;code&gt;201 Created&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;Emits &lt;code&gt;PAYMENT.CAPTURE.REFUNDED&lt;/code&gt;
&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The &lt;code&gt;refund_not_allowed&lt;/code&gt; scenario changes the second step to:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;response_status&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;422&lt;/span&gt;
&lt;span class="na"&gt;error_code&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;CAPTURE_FULLY_REFUNDED&lt;/span&gt;
&lt;span class="na"&gt;error_detail&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;The&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;capture&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;has&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;already&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;been&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;fully&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;refunded."&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Add &lt;a href="https://fetchsandbox.com/mcp" rel="noopener noreferrer"&gt;FetchSandbox MCP&lt;/a&gt; to Cursor, Claude, or another MCP-capable agent and ask:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Run paypal-payments capture_and_refund with refund_not_allowed.
Inspect my handler for the exact 422 issue. Prove it stops retries
and reconciles before reporting a failure. Return the receipt URL.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The run receipt records the request sequence and exact error contract. Your app-level assertion checks the database and UI.&lt;/p&gt;

&lt;p&gt;For CI:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;fetchsandbox run &amp;lt;sandbox-id&amp;gt; capture_and_refund &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--scenario&lt;/span&gt; refund_not_allowed &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--json&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This gives the generated integration a failure-path acceptance test instead of relying on a code reviewer to infer behavior from the happy path.&lt;/p&gt;

&lt;h2&gt;
  
  
  Can Bolt preview receive PayPal webhooks?
&lt;/h2&gt;

&lt;p&gt;Not directly. PayPal sends webhooks to a publicly reachable HTTPS endpoint, while Bolt's browser-based preview is not a public webhook destination. Deploy to Bolt Cloud, Netlify, or another host for final PayPal webhook connectivity testing.&lt;/p&gt;

&lt;p&gt;The service twin handles a different job: it lets the agent exercise provider state, API errors, and expected webhook events before that deployment. Use both layers:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;FetchSandbox for repeatable failure branches during development&lt;/li&gt;
&lt;li&gt;PayPal sandbox for final credentials and webhook connectivity&lt;/li&gt;
&lt;li&gt;Production only after both paths pass&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Does a service twin replace an API mock?
&lt;/h2&gt;

&lt;p&gt;No. A mock is useful when a component needs a shaped response. A twin is useful when correctness depends on what happened in earlier calls.&lt;/p&gt;

&lt;p&gt;Refund testing needs state:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;captured amount
- previous partial refunds
= remaining refundable amount
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That is why &lt;code&gt;CAPTURE_FULLY_REFUNDED&lt;/code&gt; is a lifecycle test rather than a fixture test.&lt;/p&gt;

&lt;h2&gt;
  
  
  Pull-request checklist
&lt;/h2&gt;

&lt;p&gt;Before merging an AI-generated PayPal refund flow, attach evidence that:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;one refund operation stores one &lt;code&gt;PayPal-Request-Id&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;timeout retries reuse that ID&lt;/li&gt;
&lt;li&gt;the handler parses &lt;code&gt;details[].issue&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;CAPTURE_FULLY_REFUNDED&lt;/code&gt; stops automatic retries&lt;/li&gt;
&lt;li&gt;local state reconciles before the UI reports success or failure&lt;/li&gt;
&lt;li&gt;the pipeline can reproduce the same 422 path&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The complete canonical guide is &lt;a href="https://fetchsandbox.com/blog/test-paypal-refunds-lovable-before-production" rel="noopener noreferrer"&gt;Test PayPal Refunds in Lovable Before Production&lt;/a&gt;. For the wider workflow, see &lt;a href="https://fetchsandbox.com/ai-app-builder-integration-testing" rel="noopener noreferrer"&gt;integration testing for AI-built apps&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>paypal</category>
      <category>lovable</category>
      <category>apitesting</category>
      <category>mcp</category>
    </item>
    <item>
      <title>Your AI builder shipped the Stripe code in an afternoon. Verifying it took a week.</title>
      <dc:creator>FetchSandbox</dc:creator>
      <pubDate>Sun, 06 Sep 2026 04:20:21 +0000</pubDate>
      <link>https://dev.to/fetchsandbox/your-ai-builder-shipped-the-stripe-code-in-an-afternoon-verifying-it-took-a-week-2o6n</link>
      <guid>https://dev.to/fetchsandbox/your-ai-builder-shipped-the-stripe-code-in-an-afternoon-verifying-it-took-a-week-2o6n</guid>
      <description>&lt;p&gt;A friend shipped a booking app last month. Lovable built the whole thing, Stripe Checkout included, in something like an afternoon. It looked finished. It &lt;em&gt;was&lt;/em&gt; finished, in the sense that you could pay and get a confirmation email.&lt;/p&gt;

&lt;p&gt;Then a customer got charged twice for one booking.&lt;/p&gt;

&lt;p&gt;Not a double-click. One payment, one &lt;code&gt;payment_intent.succeeded&lt;/code&gt;, delivered twice, because Stripe retries. The handler ran twice and created two bookings. Nobody caught it in review, and here is the uncomfortable part: the code was fine on the happy path, and nobody reviewing it had written a line of it.&lt;/p&gt;

&lt;p&gt;That is the actual shape of the problem right now. Build time collapsed. Verification time did not.&lt;/p&gt;

&lt;h2&gt;
  
  
  The generated code is not the problem
&lt;/h2&gt;

&lt;p&gt;This is where I expected to find sloppy AI output, and I didn't. The handler read exactly like something a competent developer would write on a first pass:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="nx"&gt;app&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;post&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;/webhook&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;async &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;req&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;res&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;event&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;stripe&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;webhooks&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;constructEvent&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;req&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;body&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;sig&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;secret&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;type&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;payment_intent.succeeded&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;createBooking&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;object&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;booking_id&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;sendConfirmation&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;object&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;receipt_email&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="nx"&gt;res&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;received&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Signature verified. Correct event type. Returns 200. It will pass code review and it will pass a smoke test, because the first delivery does exactly what it should.&lt;/p&gt;

&lt;p&gt;There is no idempotency key anywhere in it. The agent was never asked "what happens on the second delivery," so it never answered that question. Neither did the reviewer, because you cannot see a retry in a diff.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why the usual tools miss it
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Mocks&lt;/strong&gt; return the response you configured. A mock does not decide to retry you. That is the whole point of a mock, and it is also why it cannot catch this class of bug.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Staging&lt;/strong&gt; proves the happy path against real credentials. To see a duplicate delivery you have to wait for Stripe to genuinely retry, which means causing a timeout on purpose and waiting out a backoff. Nobody does this on a Tuesday.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;stripe trigger&lt;/code&gt;&lt;/strong&gt; fires a fresh, well-formed event. Useful, but every trigger is a &lt;em&gt;first&lt;/em&gt; delivery. The bug lives in the second one.&lt;/p&gt;

&lt;p&gt;So the failure mode is invisible to all three, and it surfaces when real money moves.&lt;/p&gt;

&lt;h2&gt;
  
  
  Force the second delivery
&lt;/h2&gt;

&lt;p&gt;What I actually wanted was a Stripe that behaves like Stripe, including the annoying parts, on demand. That is what a service twin is: stateful, so a created PaymentIntent can be read back later, and scriptable, so you can ask for the retry instead of waiting for it.&lt;/p&gt;

&lt;p&gt;FetchSandbox runs as an MCP server, so the agent that wrote the integration can drive it. In Cursor or Claude:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Run the Stripe accept_payment workflow, then re-run it under the
webhook_retries scenario. Tell me how many bookings exist afterwards
and give me the receipt URL.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;webhook_retries&lt;/code&gt; delivers each webhook once, then replays it twice with the &lt;strong&gt;same upstream event id&lt;/strong&gt; at backoff intervals, which is the behaviour that matters. A handler deduping on a per-delivery header still fails here, because that header changes on every retry while &lt;code&gt;event.id&lt;/code&gt; does not.&lt;/p&gt;

&lt;p&gt;The run came back with three bookings for one payment. Not an opinion about the code, a count.&lt;/p&gt;

&lt;p&gt;The fix is the boring one:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;alreadyProcessed&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;res&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;received&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;recordProcessed&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then re-run the same scenario and the count stays at one. That buggy-to-fixed flip is the thing worth having, because "the agent says it fixed it" and "the retry no longer double-books" are different claims.&lt;/p&gt;

&lt;h2&gt;
  
  
  The part that makes it reviewable
&lt;/h2&gt;

&lt;p&gt;Every run produces a receipt at a shareable URL: the requests in order, the state the twin held, the webhook events including the replays, and a verdict. The verdict is separate from the HTTP status on purpose, so a run can return 200 at every step and still come back unproven.&lt;/p&gt;

&lt;p&gt;That distinction is what makes this useful when you didn't write the code. You are no longer reviewing an unfamiliar diff and hoping. You are reading evidence that a specific failure was forced and handled. Put the URL in the PR.&lt;/p&gt;

&lt;p&gt;The same run works from the CLI when you want the pipeline to enforce it rather than a person remembering:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;fetchsandbox run &amp;lt;sandbox-id&amp;gt; accept_payment &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--scenario&lt;/span&gt; webhook_retries &lt;span class="nt"&gt;--json&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Local proof while building, the same check gating merges. The failure branch becomes a required step instead of a thing you meant to test.&lt;/p&gt;

&lt;h2&gt;
  
  
  Beyond Stripe
&lt;/h2&gt;

&lt;p&gt;Payments are where it bites hardest, but the pattern is identical anywhere an integration has a lifecycle. Paddle, Resend, Twilio, Clerk, Descope, and AgentMail have curated workflows and failure scenarios; the wider catalogue covers 50+ APIs generated from their OpenAPI specs. Same loop each time: run the flow, force the failure, keep the receipt.&lt;/p&gt;

&lt;p&gt;Setup is &lt;a href="https://fetchsandbox.com/mcp" rel="noopener noreferrer"&gt;one MCP config block&lt;/a&gt;, and there is more on the &lt;a href="https://fetchsandbox.com/ai-app-builder-integration-testing" rel="noopener noreferrer"&gt;approach for AI-built apps&lt;/a&gt; if you want the longer version.&lt;/p&gt;

&lt;h2&gt;
  
  
  The actual question
&lt;/h2&gt;

&lt;p&gt;I keep meeting people whose apps went from idea to paying customers in a weekend, and whose integration testing story is "staging looked fine."&lt;/p&gt;

&lt;p&gt;So, genuinely: what are you doing here? Is anyone verifying the retry, the dropped webhook, and the mid-flow auth failure before production, or is the honest answer that you find out from a support ticket? I am more interested in what people actually do than what we all agree we should do.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>webdev</category>
      <category>testing</category>
      <category>stripe</category>
    </item>
    <item>
      <title>undefined === undefined — the auth bypass your AI wrote into your checkout</title>
      <dc:creator>FetchSandbox</dc:creator>
      <pubDate>Thu, 13 Aug 2026 17:40:08 +0000</pubDate>
      <link>https://dev.to/fetchsandbox/undefined-undefined-the-auth-bypass-your-ai-wrote-into-your-checkout-4j9g</link>
      <guid>https://dev.to/fetchsandbox/undefined-undefined-the-auth-bypass-your-ai-wrote-into-your-checkout-4j9g</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;TL;DR&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;An auth check that uses &lt;code&gt;===&lt;/code&gt; where both sides can be &lt;code&gt;undefined&lt;/code&gt; is fail-open — a missing config turns it from locked to open.&lt;/li&gt;
&lt;li&gt;The one secret missing from &lt;code&gt;.env.example&lt;/code&gt; was the internal auth token. Unset in a deploy, &lt;code&gt;undefined === undefined&lt;/code&gt; let every anonymous request through.&lt;/li&gt;
&lt;li&gt;Default to deny: validate types, reject empty, compare in constant time, and write the test that sends no credentials with the env var unset.&lt;/li&gt;
&lt;/ul&gt;
&lt;/blockquote&gt;

&lt;p&gt;The coding agent handed me an auth middleware. It passed review. I almost shipped it.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;protect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;req&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;res&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;next&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;req&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;x-internal-token&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="nx"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;env&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;INTERNAL_API_TOKEN&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;next&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;requireAuth&lt;/span&gt;&lt;span class="p"&gt;()(&lt;/span&gt;&lt;span class="nx"&gt;req&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;res&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;next&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Four lines, and it reads fine. An internal service calls with a shared token and skips the login flow; everyone else falls through to real auth. Then I checked what happens when the token isn't set.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the check actually did
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;INTERNAL_API_TOKEN&lt;/code&gt; was the one secret missing from &lt;code&gt;.env.example&lt;/code&gt;. Every other key was there — Stripe, Clerk, Paddle — but not this one. So on a deploy where nobody thought to set it, &lt;code&gt;process.env.INTERNAL_API_TOKEN&lt;/code&gt; is &lt;code&gt;undefined&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Now a normal browser request comes in. It doesn't send an &lt;code&gt;x-internal-token&lt;/code&gt; header, so &lt;code&gt;req.headers['x-internal-token']&lt;/code&gt; is also &lt;code&gt;undefined&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;undefined === undefined&lt;/code&gt; → &lt;code&gt;true&lt;/code&gt; → &lt;code&gt;return next()&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Auth bypassed. Every anonymous request to &lt;code&gt;/checkout&lt;/code&gt; is treated as a trusted internal call. The endpoint is &lt;em&gt;safest when the token is set and wide open when it's missing&lt;/em&gt; — which is exactly backwards. A misconfigured auth check should get &lt;strong&gt;more&lt;/strong&gt; restrictive, not less.&lt;/p&gt;

&lt;p&gt;And there's a quieter problem even when the token &lt;em&gt;is&lt;/em&gt; set: &lt;code&gt;===&lt;/code&gt; on a secret isn't constant-time. It short-circuits on the first mismatched byte, so response timing leaks the token one character at a time.&lt;/p&gt;

&lt;h2&gt;
  
  
  The real check
&lt;/h2&gt;

&lt;p&gt;Fail closed:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;crypto&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;require&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;crypto&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;hasInternalToken&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;req&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;expected&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;env&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;INTERNAL_API_TOKEN&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;typeof&lt;/span&gt; &lt;span class="nx"&gt;expected&lt;/span&gt; &lt;span class="o"&gt;!==&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;string&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nx"&gt;expected&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="c1"&gt;// no token configured → deny&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;provided&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;req&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;x-internal-token&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;];&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;typeof&lt;/span&gt; &lt;span class="nx"&gt;provided&lt;/span&gt; &lt;span class="o"&gt;!==&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;string&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nx"&gt;provided&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;a&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;Buffer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;from&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;provided&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="nx"&gt;b&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;Buffer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;from&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;expected&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;a&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt; &lt;span class="o"&gt;!==&lt;/span&gt; &lt;span class="nx"&gt;b&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;crypto&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;timingSafeEqual&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;a&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;b&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;protect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;req&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;res&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;next&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;hasInternalToken&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;req&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;next&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;requireAuth&lt;/span&gt;&lt;span class="p"&gt;()(&lt;/span&gt;&lt;span class="nx"&gt;req&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;res&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;next&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;No configured token means no bypass. An empty or missing header means no bypass. The compare is constant-time, and length is checked first (&lt;code&gt;timingSafeEqual&lt;/code&gt; throws on length mismatch). Then add &lt;code&gt;INTERNAL_API_TOKEN&lt;/code&gt; to &lt;code&gt;.env.example&lt;/code&gt; so it can never be silently omitted again.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why agents write it this way
&lt;/h2&gt;

&lt;p&gt;The happy path looks correct. Token set, caller sends the right one, &lt;code&gt;x === y&lt;/code&gt; is &lt;code&gt;true&lt;/code&gt; — review sees a plausible auth check and moves on. The model optimizes for "works when configured," and &lt;code&gt;===&lt;/code&gt; fails open by default. It never had to reason about the unset case, because the unset case isn't in the example the code was written against.&lt;/p&gt;

&lt;p&gt;Humans do this too — the fail-open default, the &lt;code&gt;===&lt;/code&gt; on a secret, the config that's "obviously" always set. Agents just do it faster and ship it greener.&lt;/p&gt;

&lt;h2&gt;
  
  
  Fail closed, and prove it
&lt;/h2&gt;

&lt;p&gt;The tell isn't in the code. It's in the test you didn't write. Send a request with &lt;strong&gt;no credentials&lt;/strong&gt; and the env var &lt;strong&gt;unset&lt;/strong&gt;, and assert it's rejected:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;delete&lt;/span&gt; &lt;span class="nx"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;env&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;INTERNAL_API_TOKEN&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;res&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;request&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;app&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;post&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;/checkout&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;send&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;seats&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;
&lt;span class="nf"&gt;expect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;res&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;status&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nx"&gt;not&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;toBe&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt; &lt;span class="c1"&gt;// must be 401/302 — not "welcome in"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If an unauthenticated request with no token configured gets a &lt;code&gt;200&lt;/code&gt;, your auth is open — and no amount of green on the happy path will tell you. The bug lives entirely in the case the tests never exercise.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I changed in the habit
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Auth defaults to deny.&lt;/strong&gt; Every branch that grants access starts from "no."&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Validate both sides&lt;/strong&gt; — type is string, length &amp;gt; 0. &lt;code&gt;undefined === undefined&lt;/code&gt; is not authentication.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Constant-time compare&lt;/strong&gt; for any secret (&lt;code&gt;crypto.timingSafeEqual&lt;/code&gt;), never &lt;code&gt;===&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Write the misconfigured-env test&lt;/strong&gt; — unset the token, send nothing, assert rejected.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Every secret the code reads goes in &lt;code&gt;.env.example&lt;/code&gt;.&lt;/strong&gt; A missing one isn't a config gap; here it was the whole lock.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;I found this the way I find most of them: reproduced it on the real code — an anonymous request returned &lt;code&gt;200&lt;/code&gt; — applied the fix, and confirmed it flipped to a &lt;code&gt;302&lt;/code&gt;, with FetchSandbox. The bug was never in the happy path. It was in the case the code never had to handle.&lt;/p&gt;

&lt;p&gt;Which is the rule every auth check should start from: &lt;strong&gt;fail closed.&lt;/strong&gt;&lt;/p&gt;

</description>
      <category>security</category>
      <category>node</category>
      <category>webdev</category>
      <category>ai</category>
    </item>
    <item>
      <title>Green passed. The fix granted zero seats.</title>
      <dc:creator>FetchSandbox</dc:creator>
      <pubDate>Tue, 11 Aug 2026 14:05:17 +0000</pubDate>
      <link>https://dev.to/fetchsandbox/green-passed-the-fix-granted-zero-seats-3n4m</link>
      <guid>https://dev.to/fetchsandbox/green-passed-the-fix-granted-zero-seats-3n4m</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;TL;DR&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;"Did the bug stop?" is a weak invariant. An over-suppressing fix — grant nothing — also stops the bug.&lt;/li&gt;
&lt;li&gt;Assert the exact correct end state: after N identical deliveries, seats == the purchased amount. Not more. Not zero.&lt;/li&gt;
&lt;li&gt;A verifier that cannot reject a deliberately broken fix is not a verifier. It is a green light.&lt;/li&gt;
&lt;/ul&gt;
&lt;/blockquote&gt;

&lt;p&gt;The coding agent handed me a patch. CI went green. I almost merged it.&lt;/p&gt;

&lt;p&gt;The original bug was ugly and familiar. A Stripe webhook handler granted seats on &lt;code&gt;invoice.paid&lt;/code&gt; (or whatever your equivalent is — &lt;code&gt;checkout.session.completed&lt;/code&gt;, &lt;code&gt;customer.subscription.updated&lt;/code&gt;, pick your poison). It was not idempotent. Stripe retried. The handler granted again.&lt;/p&gt;

&lt;p&gt;A five-seat purchase became 5, then 10, then 15. Same event. Same customer. Three deliveries.&lt;/p&gt;

&lt;p&gt;That is the bug everyone warns you about. At-least-once delivery. Dedupe on event id. &lt;code&gt;INSERT … ON CONFLICT DO NOTHING&lt;/code&gt;. You have heard the sermon.&lt;/p&gt;

&lt;p&gt;So I asked the agent to fix it. It wrote a patch. My verifier ran the retry series and graded the result &lt;strong&gt;green&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Then I looked at the seats.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;delivery     buggy handler     agent "fix"      what we wanted
1            5                 0                5
2            10                0                5
3            15                0                5
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The count had stopped growing. True. It had also stopped granting.&lt;/p&gt;

&lt;p&gt;Customers who paid got nothing. Including on the first, legitimate delivery. The bug was "fixed" the way you fix a leaky pipe by shutting off the water main.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the check actually asserted
&lt;/h2&gt;

&lt;p&gt;The verifier was not stupid in a cartoon way. It did replay the webhook. It did look at a side effect. It did not just grep the diff for &lt;code&gt;idempotent&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;It asserted the thing I had complained about: &lt;em&gt;the count should not keep climbing&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;Something like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// Weak: "the bug stopped"&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;series&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;seatsAfterEachDelivery&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;handler&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt; &lt;span class="c1"&gt;// e.g. [0, 0, 0]&lt;/span&gt;

&lt;span class="nf"&gt;expect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;series&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;]).&lt;/span&gt;&lt;span class="nf"&gt;toBeLessThanOrEqual&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;series&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]);&lt;/span&gt;
&lt;span class="nf"&gt;expect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;series&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nx"&gt;size&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;toBe&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt; &lt;span class="c1"&gt;// "stable across retries"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;[5, 10, 15]&lt;/code&gt; fails. Good.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;[5, 5, 5]&lt;/code&gt; passes. Also good.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;[0, 0, 0]&lt;/code&gt; passes. That is the hole.&lt;/p&gt;

&lt;p&gt;Any monotonicity check, any "didn't grow," any "delta is zero after the first call" that never pins the &lt;em&gt;value&lt;/em&gt; will accept the over-suppressing patch. Delete the grant. Swallow the event. Return 200 and write nothing. The graph is flat. Green.&lt;/p&gt;

&lt;p&gt;If you have ever written &lt;code&gt;expect(errors).toHaveLength(0)&lt;/code&gt; and then watched someone delete the code that could error, you have met this family.&lt;/p&gt;

&lt;h2&gt;
  
  
  The real invariant
&lt;/h2&gt;

&lt;p&gt;Idempotency is not "nothing happens twice." It is "the same request leaves the system in the same correct state."&lt;/p&gt;

&lt;p&gt;For this handler, after N identical deliveries of a five-seat purchase, seats must equal &lt;strong&gt;exactly five&lt;/strong&gt;. Not fifteen. Not zero.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// Exact: after N identical deliveries, seats == purchased amount&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;series&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;seatsAfterEachDelivery&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;handler&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="nf"&gt;expect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;series&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;toEqual&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;]);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That one line rejects &lt;code&gt;[5, 10, 15]&lt;/code&gt; &lt;em&gt;and&lt;/em&gt; &lt;code&gt;[0, 0, 0]&lt;/code&gt;. It accepts &lt;code&gt;[5, 5, 5]&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;When I switched the assert, the zero-grant patch went red. The agent tried again. The patch that actually deduped — grant once, ignore the retries — produced &lt;code&gt;[5, 5, 5]&lt;/code&gt; and stayed green.&lt;/p&gt;

&lt;p&gt;Same deliveries. Same metric. Different question.&lt;/p&gt;

&lt;p&gt;"Did it stop doing the bad thing?" vs "Is the world in the state a correct implementation would leave it in?"&lt;/p&gt;

&lt;h2&gt;
  
  
  Why agents fail this way
&lt;/h2&gt;

&lt;p&gt;I do not think the model was being clever. I think it was being literal.&lt;/p&gt;

&lt;p&gt;You said: stop double-granting. The shortest path to "the number does not increase" is "the number never moves." Skip the write. Make the insert always look like a conflict. Short-circuit before &lt;code&gt;grantSeats&lt;/code&gt;. Plenty of ways to get a flat series.&lt;/p&gt;

&lt;p&gt;Humans do this too under time pressure. Feature flags that default off. &lt;code&gt;if (false)&lt;/code&gt; around the dangerous block. Tests that mock the collaborator into a no-op. Agents just do it faster, and they will happily stop at the first green.&lt;/p&gt;

&lt;p&gt;Your CI is an optimization target. If the loss function is "bug symptom gone," over-suppression is a local minimum. If the loss function is "exact end state," that minimum disappears.&lt;/p&gt;

&lt;p&gt;This is not a Stripe trivia item. It is any side-effecting handler you let an agent touch:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A refund webhook that "fixes" double-refunds by refunding &lt;code&gt;$0&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;A provisioner that "fixes" duplicate users by creating none.&lt;/li&gt;
&lt;li&gt;A retry queue that "fixes" duplicate emails by sending nothing, including the first time.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The shape is always the same. Symptom: too much. Naive fix: zero. Correct fix: once.&lt;/p&gt;

&lt;h2&gt;
  
  
  Positive control, negative control
&lt;/h2&gt;

&lt;p&gt;Here is the part I now require before I trust a verifier — mine, yours, an agent's, a CI job.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Positive control.&lt;/strong&gt; A known-good implementation must pass. For me that was the idempotent grant: &lt;code&gt;[5, 5, 5]&lt;/code&gt;. If your exact-count assert fails on a patch you already believe is right, the assert is wrong, not the patch.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Negative control.&lt;/strong&gt; A known-bad implementation must fail. I keep a deliberately broken patch around: the zero-grant one, or the original double-grant, or both.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="nf"&gt;expect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;seatsAfterEachDelivery&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;buggy&lt;/span&gt;&lt;span class="p"&gt;)).&lt;/span&gt;&lt;span class="nf"&gt;toEqual&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;]);&lt;/span&gt;
&lt;span class="c1"&gt;// fail: [5, 10, 15]&lt;/span&gt;

&lt;span class="nf"&gt;expect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;seatsAfterEachDelivery&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;zeroGrant&lt;/span&gt;&lt;span class="p"&gt;)).&lt;/span&gt;&lt;span class="nf"&gt;toEqual&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;]);&lt;/span&gt;
&lt;span class="c1"&gt;// fail: [0, 0, 0]   ← this is the control most suites skip&lt;/span&gt;

&lt;span class="nf"&gt;expect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;seatsAfterEachDelivery&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;idempotent&lt;/span&gt;&lt;span class="p"&gt;)).&lt;/span&gt;&lt;span class="nf"&gt;toEqual&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;]);&lt;/span&gt;
&lt;span class="c1"&gt;// pass&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If the zero-grant still greens, you do not have a proof. You have a check that the symptom you first noticed got quieter.&lt;/p&gt;

&lt;p&gt;I used to stop at the positive control. "The good fix passes, ship it." That is how &lt;code&gt;[0, 0, 0]&lt;/code&gt; got a green. The suite had never been shown a lying patch.&lt;/p&gt;

&lt;p&gt;This is the same idea as a mutation test, just less academic. You do not need a framework. You need one broken cousin of the fix, and a gate that refuses to call itself a verifier until that cousin fails.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I changed in the habit
&lt;/h2&gt;

&lt;p&gt;I still replay the webhook three times. That part was never the mistake. The mistake was scoring the replay with a bound instead of an equality.&lt;/p&gt;

&lt;p&gt;The checklist I use now, when an agent "fixes" a production-shaped bug:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Write the end state in English first. "After three identical &lt;code&gt;invoice.paid&lt;/code&gt; deliveries for a 5-seat price, &lt;code&gt;seat_limit == 5&lt;/code&gt;."&lt;/li&gt;
&lt;li&gt;Put that number in the assert. Not &lt;code&gt;&amp;lt;=&lt;/code&gt;. Not "unchanged after first." The number.&lt;/li&gt;
&lt;li&gt;Run the original bug. It must fail that assert.&lt;/li&gt;
&lt;li&gt;Run a spiteful fix (no-op, always-conflict, grant-zero). It must fail too.&lt;/li&gt;
&lt;li&gt;Then run the agent's patch.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Step 4 is the one I had skipped. It is also the cheapest. You can write the no-op in thirty seconds. If your suite cannot fail it, do not let the suite pass the agent.&lt;/p&gt;

&lt;p&gt;I hit this while building FetchSandbox, replaying Stripe deliveries against persistent seat state instead of a one-shot fixture.&lt;/p&gt;

&lt;p&gt;The transferable rule is not about Stripe, and it is not about my tooling. When you verify an AI-written fix, "did the bug stop?" is the wrong question. Ask whether the system is in the exact state a correct implementation would leave it in — and prove your question can still say no.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>testing</category>
      <category>webdev</category>
      <category>stripe</category>
    </item>
    <item>
      <title>You can't code-review a webhook retry</title>
      <dc:creator>FetchSandbox</dc:creator>
      <pubDate>Tue, 04 Aug 2026 17:51:20 +0000</pubDate>
      <link>https://dev.to/fetchsandbox/you-cant-code-review-a-webhook-retry-1gkk</link>
      <guid>https://dev.to/fetchsandbox/you-cant-code-review-a-webhook-retry-1gkk</guid>
      <description>&lt;p&gt;My coding agent wrote a Stripe webhook handler a few weeks ago. Signature check, event type check, fulfillment call, clean 200. The diff read like something I'd write on a good day. I approved it in about ninety seconds.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="nx"&gt;app&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;post&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;/webhooks/stripe&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;async &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;req&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;res&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;event&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;verifySignature&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;req&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;type&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;charge.succeeded&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;creditCustomer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;object&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="nx"&gt;res&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sendStatus&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Every line is correct. And the first time Stripe delivers that event twice — which it's allowed to do, and eventually will — the handler credits the customer twice.&lt;/p&gt;

&lt;h2&gt;
  
  
  Reading can't catch it
&lt;/h2&gt;

&lt;p&gt;The bug isn't on any line, so no amount of reading finds it. Each statement is fine on its own. What's wrong is an assumption living between the lines: that every event arrives exactly once. Stripe promises &lt;em&gt;at-least-once&lt;/em&gt; delivery. Retries and duplicates are documented, normal behavior.&lt;/p&gt;

&lt;p&gt;Code review checks the diff against the traffic you can imagine. I imagined one clean &lt;code&gt;charge.succeeded&lt;/code&gt;. So did the agent — the quickstart code it learned from imagined one too.&lt;/p&gt;

&lt;h2&gt;
  
  
  CI passed for the same reason
&lt;/h2&gt;

&lt;p&gt;My tests replayed the events I thought of. The fixture file had exactly one copy of each event, because I wrote the fixture file. Nobody writes the test where the same event lands twice nine minutes apart. The suite was my assumptions checking my assumptions.&lt;/p&gt;

&lt;p&gt;I spent three years on the developer platform of a very large payments company. This exact bug got past good reviewers more times than I can count. It was never a talent problem. Reading is the wrong tool for this class of bug.&lt;/p&gt;

&lt;h2&gt;
  
  
  The review that actually works
&lt;/h2&gt;

&lt;p&gt;What caught it was running the failure. Deliver the event, deliver it again, watch the customer get credited twice. Add the dedupe on the event ID, run it again, watch it hold. Ten minutes, and the review went from "looks right" to "proved right."&lt;/p&gt;

&lt;p&gt;That loop — reproduce the failure, fix it, rerun, keep the receipt — is what I built &lt;a href="https://fetchsandbox.com" rel="noopener noreferrer"&gt;FetchSandbox&lt;/a&gt; to run from your IDE over MCP. The sandbox fires the real charge lifecycle, including the duplicate delivery your fixture file doesn't have, so your agent proves the handler before your customers do.&lt;/p&gt;

&lt;p&gt;If you'd rather judge it on real code than my word, there's a webhook-dedupe bug planted in a Stripe app in our open playground, waiting to be caught: &lt;a href="https://github.com/fetchsandbox/playground" rel="noopener noreferrer"&gt;github.com/fetchsandbox/playground&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>webdev</category>
      <category>stripe</category>
      <category>ai</category>
      <category>testing</category>
    </item>
    <item>
      <title>Why Paddle's subscription.activated arrives before subscription.created</title>
      <dc:creator>FetchSandbox</dc:creator>
      <pubDate>Thu, 30 Jul 2026 13:58:08 +0000</pubDate>
      <link>https://dev.to/fetchsandbox/why-paddles-subscriptionactivated-arrives-before-subscriptioncreated-dc0</link>
      <guid>https://dev.to/fetchsandbox/why-paddles-subscriptionactivated-arrives-before-subscriptioncreated-dc0</guid>
      <description>&lt;p&gt;&lt;em&gt;You'd think events fire in the order things happen. They don't.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;I was building a Paddle integration last week. Subscription billing, nothing fancy. Customer clicks buy, Paddle handles checkout, my app gets webhooks and updates the database.&lt;/p&gt;

&lt;p&gt;The flow should be simple:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Customer completes checkout&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;subscription.created&lt;/code&gt; fires&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;subscription.activated&lt;/code&gt; fires&lt;/li&gt;
&lt;li&gt;My app inserts a row on &lt;code&gt;.created&lt;/code&gt;, updates status on &lt;code&gt;.activated&lt;/code&gt;
&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;That's what I built. It worked great in my head.&lt;/p&gt;

&lt;h2&gt;
  
  
  What actually happened
&lt;/h2&gt;

&lt;p&gt;In production, &lt;code&gt;subscription.activated&lt;/code&gt; arrived &lt;em&gt;before&lt;/em&gt; &lt;code&gt;subscription.created&lt;/code&gt; about 30% of the time.&lt;/p&gt;

&lt;p&gt;My handler did a database insert on &lt;code&gt;subscription.created&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;subscription.created&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;db&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;insert&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;subscriptions&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;values&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
    &lt;span class="na"&gt;paddleId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="na"&gt;status&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;created&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="na"&gt;customerId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;customer_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;});&lt;/span&gt;
  &lt;span class="k"&gt;break&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And an update on &lt;code&gt;subscription.activated&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;subscription.activated&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;db&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;update&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;subscriptions&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;status&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;active&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;where&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;eq&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;subscriptions&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;paddleId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
  &lt;span class="k"&gt;break&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;When &lt;code&gt;.activated&lt;/code&gt; arrived first, the update found zero rows. No error, no exception. The &lt;code&gt;WHERE&lt;/code&gt; clause just matched nothing. The update silently did nothing.&lt;/p&gt;

&lt;p&gt;Then &lt;code&gt;.created&lt;/code&gt; arrived and inserted the row with status &lt;code&gt;created&lt;/code&gt;. But the &lt;code&gt;.activated&lt;/code&gt; event was already gone. So the subscription was stuck in &lt;code&gt;created&lt;/code&gt; status forever.&lt;/p&gt;

&lt;p&gt;Customers had paid. Paddle showed them as active. My app showed them as pending. Support tickets started coming in.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why this happens
&lt;/h2&gt;

&lt;p&gt;Paddle does not guarantee webhook delivery order. Their docs mention it briefly but it's easy to miss when you're focused on the API endpoints.&lt;/p&gt;

&lt;p&gt;The events are fired from different internal services. &lt;code&gt;subscription.created&lt;/code&gt; comes from the subscription service. &lt;code&gt;subscription.activated&lt;/code&gt; comes from the billing service after payment confirmation. They are async. They race.&lt;/p&gt;

&lt;p&gt;This is not unique to Paddle either. Stripe has the same problem with &lt;code&gt;payment_intent.created&lt;/code&gt; vs &lt;code&gt;charge.succeeded&lt;/code&gt;. Most payment providers have some version of this.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix
&lt;/h2&gt;

&lt;p&gt;The handler needs to be idempotent and order-independent. Every event should be able to create or update:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;subscription.created&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;subscription.activated&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;event_type&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;subscription.activated&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt; 
    &lt;span class="p"&gt;?&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;active&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;created&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

  &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;db&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;insert&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;subscriptions&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;values&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
      &lt;span class="na"&gt;paddleId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nx"&gt;status&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="na"&gt;customerId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;customer_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;})&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;onConflictDoUpdate&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
      &lt;span class="na"&gt;target&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;subscriptions&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;paddleId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="na"&gt;set&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; 
        &lt;span class="na"&gt;status&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;sql&lt;/span&gt;&lt;span class="s2"&gt;`CASE WHEN &lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;status&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; = 'active' THEN 'active' ELSE &lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;subscriptions&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;status&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; END`&lt;/span&gt;
      &lt;span class="p"&gt;},&lt;/span&gt;
    &lt;span class="p"&gt;});&lt;/span&gt;
  &lt;span class="k"&gt;break&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The key parts:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Both events can create the row if it doesn't exist&lt;/li&gt;
&lt;li&gt;On conflict, &lt;code&gt;active&lt;/code&gt; always wins over &lt;code&gt;created&lt;/code&gt; regardless of arrival order&lt;/li&gt;
&lt;li&gt;No silent failures, no missing updates&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Testing this is the real problem
&lt;/h2&gt;

&lt;p&gt;The ordering bug is easy to fix once you know about it. The hard part is reproducing it during development.&lt;/p&gt;

&lt;p&gt;You can't control the order Paddle sends webhooks. You can't make &lt;code&gt;.activated&lt;/code&gt; arrive first on demand. In testing you might run through the flow 20 times and the events always arrive in order. Then in production with real network latency and load, they don't.&lt;/p&gt;

&lt;p&gt;I ended up testing this by sending the webhook events manually in the wrong order against a local sandbox. &lt;code&gt;activated&lt;/code&gt; first, then &lt;code&gt;created&lt;/code&gt;. Immediately saw the bug. Fixed it in 10 minutes.&lt;/p&gt;

&lt;p&gt;The debugging in production took 4 hours.&lt;/p&gt;

&lt;p&gt;If you're integrating Paddle or any payment provider with webhooks, test with events arriving in every possible order. Not just the happy path order from the docs.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://fetchsandbox.com/docs/paddle?utm_source=devto&amp;amp;utm_medium=blog&amp;amp;utm_campaign=paddle-activated-before-created" rel="noopener noreferrer"&gt;Test Paddle webhook ordering in a sandbox →&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The takeaway
&lt;/h2&gt;

&lt;p&gt;Webhook events are not a queue. They are concurrent messages from different services that happen to be about the same thing. Your handler has to treat every event as potentially the first one it sees for that resource.&lt;/p&gt;

&lt;p&gt;If your handler has an insert for one event type and an update for another, you have this bug. You just haven't hit it in production yet.&lt;/p&gt;

</description>
      <category>paddle</category>
      <category>webhooks</category>
      <category>api</category>
      <category>testing</category>
    </item>
    <item>
      <title>Webhook handlers that silently return 402: the event ordering bug most test suites miss</title>
      <dc:creator>FetchSandbox</dc:creator>
      <pubDate>Tue, 21 Jul 2026 17:40:10 +0000</pubDate>
      <link>https://dev.to/fetchsandbox/webhook-handlers-that-silently-return-402-the-event-ordering-bug-most-test-suites-miss-4f0e</link>
      <guid>https://dev.to/fetchsandbox/webhook-handlers-that-silently-return-402-the-event-ordering-bug-most-test-suites-miss-4f0e</guid>
      <description>&lt;p&gt;&lt;em&gt;How to prove a handler is order-safe using permutation fuzzing&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;You're testing your Paddle webhook handler. You've got a test for subscription.created and subscription.activated, and maybe another one where you send them out of order. Looks good. All green. But weeks later, a customer hits a 402 on subscription.activated. No error in your logs. No alert. Just a failed charge and a confused user.&lt;/p&gt;

&lt;p&gt;The problem isn't your test suite, it's the order of events.&lt;/p&gt;

&lt;p&gt;A Paddle webhook handler that does an INSERT on subscription.created and an UPDATE on subscription.activated will silently return 402 when subscription.activated arrives first. The row doesn't exist yet. No error is raised. But if you send them in the "happy path" order, it works. Your test suite never sees the failure because it only runs one sequence. The bug is invisible until it hits production.&lt;/p&gt;

&lt;p&gt;To prove a handler is order-safe, we built a permutation fuzzer in fetchsandbox MCP. It fires the same events in every permutation, including duplicates, and checks if the final state is the same in all of them. If it is, the handler is order-safe. If not, it returns the exact sequence that breaks it.&lt;/p&gt;

&lt;p&gt;Here's what it found for the Paddle handler above:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;When you send subscription.activated before subscription.created: the handler returns a 402.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;When you send them in the "happy path" order: it returns a 200.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;The fuzzer returns order_independent=False and shows you the diverging sequence.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;After fixing the handler to use an upsert instead of separate INSERT and UPDATE, the fuzzer returns order_independent=True across every permutation and duplicate.&lt;/p&gt;

&lt;p&gt;If you test API integrations in sandboxes constantly, you've hit this one too. I run it in Cursor every time I test a new webhook integration.&lt;/p&gt;

&lt;p&gt;The fuzzer is in fetchsandbox MCP. It reads final state after each permutation, and reports order_independent=True only when every ordering converges to the same final state.&lt;/p&gt;

&lt;p&gt;If you're trying to find webhook ordering bugs, or prove a handler is order-safe, this is the test you didn't know you needed.&lt;/p&gt;

&lt;p&gt;Read the full spec for Paddle webhook testing at &lt;a href="https://fetchsandbox.com/docs/paddle" rel="noopener noreferrer"&gt;Paddle sandbox&lt;/a&gt;. Learn how fetchsandbox MCP works at &lt;a href="https://fetchsandbox.com/mcp" rel="noopener noreferrer"&gt;fetchsandbox MCP&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>webhooks</category>
      <category>testing</category>
      <category>devtools</category>
      <category>api</category>
    </item>
    <item>
      <title>We planted a Descope privilege-escalation bug — can your coding agent catch it?</title>
      <dc:creator>FetchSandbox</dc:creator>
      <pubDate>Fri, 10 Jul 2026 18:39:39 +0000</pubDate>
      <link>https://dev.to/fetchsandbox/we-planted-a-descope-privilege-escalation-bug-can-your-coding-agent-catch-it-18id</link>
      <guid>https://dev.to/fetchsandbox/we-planted-a-descope-privilege-escalation-bug-can-your-coding-agent-catch-it-18id</guid>
      <description>&lt;p&gt;We open-sourced a repo with a bug planted on purpose: &lt;a href="https://github.com/fetchsandbox/playground" rel="noopener noreferrer"&gt;github.com/fetchsandbox/playground&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;The bug is in &lt;code&gt;apps/descope&lt;/code&gt; — a small FastAPI "Agent Gateway" where AI agents exchange a Descope access key for a scoped session. The flaw: the exchange endpoint trusts whatever scopes the client requests. A read-only agent key can ask for &lt;code&gt;users:write&lt;/code&gt; and get it.&lt;/p&gt;

&lt;p&gt;This is the privilege-escalation shape that ships constantly in agentic auth. The token exchange returns 200, the happy path works, and nothing ever compares the granted scope against what the key was actually granted.&lt;/p&gt;

&lt;h2&gt;
  
  
  The 20-minute test drive
&lt;/h2&gt;

&lt;p&gt;Two tasks, both run from your IDE (Cursor, Claude Code, or any MCP-capable editor). No Descope account, no API keys — the workflows run against FetchSandbox's hosted Descope sandbox over MCP, and each app ships a &lt;code&gt;.mcp.json&lt;/code&gt; already wired.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Task 1 — greenfield.&lt;/strong&gt; &lt;code&gt;apps/descope-onboarding&lt;/code&gt; is a tiny notes app with placeholder auth. Ask your agent to add Descope OTP sign-up, but with one constraint: prove the OTP + session flow in the sandbox &lt;em&gt;before&lt;/em&gt; writing any code, then propose the diff.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;./fetchsandbox I'm adding Descope OTP sign-up to this app — prove the Descope
OTP + session flow in the sandbox before writing any code, then propose the
diff. I'll decide whether to apply.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Task 2 — brownfield.&lt;/strong&gt; Point the agent at the Agent Gateway and ask it to audit:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;./fetchsandbox our agent access-key exchange might be handing out more scope
than the key was granted — audit the descope agentic auth
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  What a real catch looks like
&lt;/h2&gt;

&lt;p&gt;Static review is not the bar. "This looks vulnerable" is a guess. The bar is:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;the escalation is &lt;strong&gt;reproduced&lt;/strong&gt; — read-only key in, &lt;code&gt;users:write&lt;/code&gt; session out&lt;/li&gt;
&lt;li&gt;the proof shows &lt;strong&gt;buggy vs fixed&lt;/strong&gt; on actual Descope routes (&lt;code&gt;/v1/...&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;there's a &lt;strong&gt;receipt URL&lt;/strong&gt; — an openable run trace, not a claim in chat&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Auth bugs live in the lifecycle: replayed magic links, expired sessions, JWTs that were decoded instead of verified, scope grants nobody re-checked. Mocks return clean 200s for all of these. That's the gap the playground is designed to expose.&lt;/p&gt;

&lt;h2&gt;
  
  
  The other planted bugs
&lt;/h2&gt;

&lt;p&gt;If Descope isn't your stack, the repo also has broken Stripe (webhook dedup keyed on the wrong header — events processed 2–3×), Resend (bounces silently dropped, users stay "active"), Clerk (session validation skipped on one endpoint), and a few more. Each app is ~50–150 lines of Python.&lt;/p&gt;

&lt;h2&gt;
  
  
  The part where you tell us it fell flat
&lt;/h2&gt;

&lt;p&gt;Findings go back as a PR — there's a &lt;code&gt;FINDINGS_TEMPLATE.md&lt;/code&gt; in the repo. Paste your agent's session, the receipt URLs, and your honest reaction. Merged PRs show up on your GitHub contribution graph.&lt;/p&gt;

&lt;p&gt;We explicitly want the failure reports: MCP wouldn't connect, the agent caught nothing, the proof felt fake. An honest "it didn't work" beats a polite green checkmark.&lt;/p&gt;

&lt;p&gt;Start here: &lt;a href="https://github.com/fetchsandbox/playground/blob/main/TESTING.md" rel="noopener noreferrer"&gt;TESTING.md&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>descope</category>
      <category>security</category>
      <category>testing</category>
      <category>opensource</category>
    </item>
    <item>
      <title>How to test Descope auth without real keys</title>
      <dc:creator>FetchSandbox</dc:creator>
      <pubDate>Wed, 08 Jul 2026 14:04:28 +0000</pubDate>
      <link>https://dev.to/fetchsandbox/how-to-test-descope-auth-without-real-keys-4jib</link>
      <guid>https://dev.to/fetchsandbox/how-to-test-descope-auth-without-real-keys-4jib</guid>
      <description>&lt;p&gt;Title options:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;How to test Descope auth without real keys&lt;/li&gt;
&lt;li&gt;How to test Descope passwordless auth without a project&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Descope is easy to start with, but the part I would test first is not the OTP screen or the magic-link email.&lt;/p&gt;

&lt;p&gt;It is what your app does with the session JWT after Descope returns it.&lt;/p&gt;

&lt;p&gt;FetchSandbox has a Descope sandbox for the flows most apps wire up first:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;email OTP signup&lt;/li&gt;
&lt;li&gt;email magic-link sign-in&lt;/li&gt;
&lt;li&gt;session refresh&lt;/li&gt;
&lt;li&gt;agent/access-key exchange&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;No real Descope project. No real keys. No inbox setup.&lt;/p&gt;

&lt;p&gt;The useful part is that the sandbox models the failure cases too: wrong OTP, expired session, replayed magic link, and the big one: accepting a forged session JWT because the app decoded it instead of verifying it.&lt;/p&gt;

&lt;p&gt;Here is the concrete flow I would test.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Send a magic link:
&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight http"&gt;&lt;code&gt;&lt;span class="err"&gt;POST /v1/auth/magiclink/signin/email
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ol&gt;
&lt;li&gt;Verify the one-time token:
&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight http"&gt;&lt;code&gt;&lt;span class="err"&gt;POST /v1/auth/magiclink/verify
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The sandbox returns the shape your app should care about:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"sessionJwt"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"..."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"refreshJwt"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"..."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"user"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"userId"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"U2..."&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ol&gt;
&lt;li&gt;Call your protected route with the &lt;code&gt;sessionJwt&lt;/code&gt;.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;This is where the bug usually lives.&lt;/p&gt;

&lt;p&gt;Bad handlers do something like:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;claims&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;jwt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;decode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;sessionJwt&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;claims&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;role&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;admin&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nf"&gt;allow&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That accepts whatever claims are inside the token. A forged &lt;code&gt;alg: none&lt;/code&gt; token can become &lt;code&gt;role=admin&lt;/code&gt; if the handler never checks the signature.&lt;/p&gt;

&lt;p&gt;The fixed version verifies the session JWT against Descope's JWKS, or uses Descope's SDK validation path. It should reject &lt;code&gt;alg: none&lt;/code&gt;, reject bad signatures, cache JWKS, and refresh on &lt;code&gt;kid&lt;/code&gt; mismatch.&lt;/p&gt;

&lt;p&gt;The sandbox route for this bug is meant to prove that difference:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;forged unsigned token -&amp;gt; &lt;code&gt;401&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;valid signed session token -&amp;gt; &lt;code&gt;200&lt;/code&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That is the whole point. Do not just test "magic link works." Test that the session you trust is actually verified.&lt;/p&gt;

&lt;p&gt;You can also test refresh behavior:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight http"&gt;&lt;code&gt;&lt;span class="err"&gt;POST /v1/auth/refresh
GET /v1/auth/me
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If refresh returns &lt;code&gt;401&lt;/code&gt;, send the user back through sign-in. Do not keep rendering authenticated UI from a decoded-but-unverified token.&lt;/p&gt;

&lt;p&gt;Docs are here if you want the exact Descope sandbox flows: &lt;a href="https://fetchsandbox.com/docs/descope" rel="noopener noreferrer"&gt;https://fetchsandbox.com/docs/descope&lt;/a&gt;&lt;/p&gt;

</description>
      <category>auth</category>
      <category>api</category>
      <category>security</category>
      <category>descope</category>
    </item>
    <item>
      <title>Our CI mocked Stripe. Production still broke on step two.</title>
      <dc:creator>FetchSandbox</dc:creator>
      <pubDate>Tue, 30 Jun 2026 14:23:58 +0000</pubDate>
      <link>https://dev.to/fetchsandbox/our-ci-mocked-stripe-production-still-broke-on-step-two-475d</link>
      <guid>https://dev.to/fetchsandbox/our-ci-mocked-stripe-production-still-broke-on-step-two-475d</guid>
      <description>&lt;p&gt;Green CI. Broken staging. Sound familiar?&lt;/p&gt;

&lt;p&gt;Our pipeline mocked &lt;code&gt;POST /v1/payment_intents&lt;/code&gt; and asserted &lt;code&gt;200&lt;/code&gt;. Merge button enabled. First real checkout in staging failed because the handler never stored the PaymentIntent ID — step two of the webhook path read &lt;code&gt;undefined&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Mocks proved &lt;strong&gt;shape&lt;/strong&gt;. Not &lt;strong&gt;lifecycle&lt;/strong&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  What unit tests miss
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;CI check&lt;/th&gt;
&lt;th&gt;What it proves&lt;/th&gt;
&lt;th&gt;What it skips&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Mocked &lt;code&gt;fetch()&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Request JSON looks right&lt;/td&gt;
&lt;td&gt;Step 2 cannot read step 1's ID&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Snapshot tests&lt;/td&gt;
&lt;td&gt;Handler code unchanged&lt;/td&gt;
&lt;td&gt;Webhook branch never runs&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Manual smoke doc&lt;/td&gt;
&lt;td&gt;Someone remembers sometimes&lt;/td&gt;
&lt;td&gt;Agents ship faster than the checklist&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The bug class: &lt;strong&gt;IDs do not chain&lt;/strong&gt; across steps. Webhooks fire on mutations your mock never triggers.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we run now — MCP locally, same workflow in CI
&lt;/h2&gt;

&lt;p&gt;Before an agent touches payment code, we prove the provider flow in the IDE via &lt;strong&gt;FetchSandbox MCP&lt;/strong&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;quickrun stripe / accept_payment
→ create PaymentIntent
→ confirm / capture path
→ read back state
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That is the contract the integration must satisfy — not a paragraph in the PR description.&lt;/p&gt;

&lt;p&gt;Then CI runs the &lt;strong&gt;same curated workflow&lt;/strong&gt; headless:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="c1"&gt;# .github/workflows/api-integration.yml&lt;/span&gt;
&lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Prove Stripe integration before deploy&lt;/span&gt;
  &lt;span class="na"&gt;run&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;npx fetchsandbox run stripe --all --json&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Exit code &lt;strong&gt;0&lt;/strong&gt; → workflows passed, merge allowed.&lt;br&gt;&lt;br&gt;
Exit code &lt;strong&gt;1&lt;/strong&gt; → named step failed (&lt;code&gt;create_payment_intent&lt;/code&gt;, webhook reconcile, etc.) — PR blocked.&lt;/p&gt;

&lt;p&gt;No staging dependency. No IP whitelist. No "works on my machine."&lt;/p&gt;

&lt;h2&gt;
  
  
  Why MCP belongs in the loop
&lt;/h2&gt;

&lt;p&gt;Coding agents edit integration code constantly. Without a runnable proof:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The agent "fixes" the handler and moves on&lt;/li&gt;
&lt;li&gt;CI mocks still pass&lt;/li&gt;
&lt;li&gt;Production breaks on the path the agent never exercised&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;MCP gives the agent (and you) a &lt;strong&gt;receipt&lt;/strong&gt;: sandbox timeline, step statuses, real provider-shaped IDs from the run — before opening the PR.&lt;/p&gt;

&lt;p&gt;CI is the gate that keeps that proof from regressing on the next agent session.&lt;/p&gt;

&lt;h2&gt;
  
  
  What a workflow run actually checks
&lt;/h2&gt;

&lt;p&gt;Not "does POST return 200 once."&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Create a resource → mutate it → &lt;strong&gt;GET the same ID back&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;Trigger the webhook branch when state changes&lt;/li&gt;
&lt;li&gt;Fail with a &lt;strong&gt;named step&lt;/strong&gt; when auth or transitions break&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That is integration testing — not HTTP cosplay.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we stopped doing
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Treating mocked unit tests as "integration covered"&lt;/li&gt;
&lt;li&gt;Running only manual smoke before deploy&lt;/li&gt;
&lt;li&gt;Letting agents merge Stripe/Resend/Clerk changes without a workflow receipt&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Try it
&lt;/h2&gt;

&lt;p&gt;Full write-up: &lt;a href="https://fetchsandbox.com/api-integration-testing-in-ci" rel="noopener noreferrer"&gt;API integration testing in CI&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;MCP + agent workflows: &lt;a href="https://fetchsandbox.com/agent-api-workflow-testing" rel="noopener noreferrer"&gt;Agent API workflow testing&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Install MCP in Cursor/Claude → run a Stripe workflow locally → paste the same &lt;code&gt;npx fetchsandbox run&lt;/code&gt; step into Actions.&lt;/p&gt;




&lt;p&gt;&lt;a href="https://fetchsandbox.com" rel="noopener noreferrer"&gt;fetchsandbox.com&lt;/a&gt; · &lt;a href="https://fetchsandbox.com/mcp" rel="noopener noreferrer"&gt;MCP setup&lt;/a&gt;&lt;/p&gt;

</description>
      <category>api</category>
      <category>devops</category>
      <category>mcp</category>
      <category>testing</category>
    </item>
  </channel>
</rss>
