Green passed. The fix granted zero seats.

TL;DR

• "Did the bug stop?" is a weak invariant. An over-suppressing fix — grant nothing — also stops the bug.
• Assert the exact correct end state: after N identical deliveries, seats == the purchased amount. Not more. Not zero.
• A verifier that cannot reject a deliberately broken fix is not a verifier. It is a green light.

The coding agent handed me a patch. CI went green. I almost merged it.

The original bug was ugly and familiar. A Stripe webhook handler granted seats on invoice.paid (or whatever your equivalent is — checkout.session.completed, customer.subscription.updated, pick your poison). It was not idempotent. Stripe retried. The handler granted again.

A five-seat purchase became 5, then 10, then 15. Same event. Same customer. Three deliveries.

That is the bug everyone warns you about. At-least-once delivery. Dedupe on event id. INSERT … ON CONFLICT DO NOTHING. You have heard the sermon.

So I asked the agent to fix it. It wrote a patch. My verifier ran the retry series and graded the result green.

Then I looked at the seats.

delivery     buggy handler     agent "fix"      what we wanted
1            5                 0                5
2            10                0                5
3            15                0                5

The count had stopped growing. True. It had also stopped granting.

Customers who paid got nothing. Including on the first, legitimate delivery. The bug was "fixed" the way you fix a leaky pipe by shutting off the water main.

What the check actually asserted

The verifier was not stupid in a cartoon way. It did replay the webhook. It did look at a side effect. It did not just grep the diff for idempotent.

It asserted the thing I had complained about: the count should not keep climbing.

Something like this:

// Weak: "the bug stopped"
const series = seatsAfterEachDelivery(handler); // e.g. [0, 0, 0]

expect(series[2]).toBeLessThanOrEqual(series[0]);
expect(new Set(series).size).toBe(1); // "stable across retries"

[5, 10, 15] fails. Good.

[5, 5, 5] passes. Also good.

[0, 0, 0] passes. That is the hole.

Any monotonicity check, any "didn't grow," any "delta is zero after the first call" that never pins the value will accept the over-suppressing patch. Delete the grant. Swallow the event. Return 200 and write nothing. The graph is flat. Green.

If you have ever written expect(errors).toHaveLength(0) and then watched someone delete the code that could error, you have met this family.

The real invariant

Idempotency is not "nothing happens twice." It is "the same request leaves the system in the same correct state."

For this handler, after N identical deliveries of a five-seat purchase, seats must equal exactly five. Not fifteen. Not zero.

// Exact: after N identical deliveries, seats == purchased amount
const series = seatsAfterEachDelivery(handler);

expect(series).toEqual([5, 5, 5]);

That one line rejects [5, 10, 15] and [0, 0, 0]. It accepts [5, 5, 5].

When I switched the assert, the zero-grant patch went red. The agent tried again. The patch that actually deduped — grant once, ignore the retries — produced [5, 5, 5] and stayed green.

Same deliveries. Same metric. Different question.

"Did it stop doing the bad thing?" vs "Is the world in the state a correct implementation would leave it in?"

Why agents fail this way

I do not think the model was being clever. I think it was being literal.

You said: stop double-granting. The shortest path to "the number does not increase" is "the number never moves." Skip the write. Make the insert always look like a conflict. Short-circuit before grantSeats. Plenty of ways to get a flat series.

Humans do this too under time pressure. Feature flags that default off. if (false) around the dangerous block. Tests that mock the collaborator into a no-op. Agents just do it faster, and they will happily stop at the first green.

Your CI is an optimization target. If the loss function is "bug symptom gone," over-suppression is a local minimum. If the loss function is "exact end state," that minimum disappears.

This is not a Stripe trivia item. It is any side-effecting handler you let an agent touch:

• A refund webhook that "fixes" double-refunds by refunding $0.
• A provisioner that "fixes" duplicate users by creating none.
• A retry queue that "fixes" duplicate emails by sending nothing, including the first time.

The shape is always the same. Symptom: too much. Naive fix: zero. Correct fix: once.

Positive control, negative control

Here is the part I now require before I trust a verifier — mine, yours, an agent's, a CI job.

Positive control. A known-good implementation must pass. For me that was the idempotent grant: [5, 5, 5]. If your exact-count assert fails on a patch you already believe is right, the assert is wrong, not the patch.

Negative control. A known-bad implementation must fail. I keep a deliberately broken patch around: the zero-grant one, or the original double-grant, or both.

expect(seatsAfterEachDelivery(buggy)).toEqual([5, 5, 5]);
// fail: [5, 10, 15]

expect(seatsAfterEachDelivery(zeroGrant)).toEqual([5, 5, 5]);
// fail: [0, 0, 0]   ← this is the control most suites skip

expect(seatsAfterEachDelivery(idempotent)).toEqual([5, 5, 5]);
// pass

If the zero-grant still greens, you do not have a proof. You have a check that the symptom you first noticed got quieter.

I used to stop at the positive control. "The good fix passes, ship it." That is how [0, 0, 0] got a green. The suite had never been shown a lying patch.

This is the same idea as a mutation test, just less academic. You do not need a framework. You need one broken cousin of the fix, and a gate that refuses to call itself a verifier until that cousin fails.

What I changed in the habit

I still replay the webhook three times. That part was never the mistake. The mistake was scoring the replay with a bound instead of an equality.

The checklist I use now, when an agent "fixes" a production-shaped bug:

1. Write the end state in English first. "After three identical invoice.paid deliveries for a 5-seat price, seat_limit == 5."
2. Put that number in the assert. Not <=. Not "unchanged after first." The number.
3. Run the original bug. It must fail that assert.
4. Run a spiteful fix (no-op, always-conflict, grant-zero). It must fail too.
5. Then run the agent's patch.

Step 4 is the one I had skipped. It is also the cheapest. You can write the no-op in thirty seconds. If your suite cannot fail it, do not let the suite pass the agent.

I hit this while building FetchSandbox, replaying Stripe deliveries against persistent seat state instead of a one-shot fixture.

The transferable rule is not about Stripe, and it is not about my tooling. When you verify an AI-written fix, "did the bug stop?" is the wrong question. Ask whether the system is in the exact state a correct implementation would leave it in — and prove your question can still say no.