We let an AI do the QA on a payment webhook. It did a good job. That was the problem.

"Let the AI write the tests" is becoming the default answer to "we don't have time for QA". We wanted to know what actually goes wrong when you do that, so we tried to catch an AI breaking QA on a small piece of payment code.

It mostly didn't break anything. What it did instead is the part worth writing about.

The code

This is a minimal reproduction of a webhook handler from one of our own side projects, a booking app that takes card payments. The real one is async and talks to a database. This one is plain Python, and it has the same three bugs:

def handle_webhook(payments, event):
    if not event.get("order_id") and not event.get("payment_id"):
        return {"result": "ignored", "reason": "missing ids"}
    payment = find_payment(payments, event)   # matches order_id OR payment_id; raises Conflict on two
    if payment is None:
        return {"result": "ignored", "reason": "no matching payment"}
    if event.get("payment_id") and not payment["payment_id"]:
        payment["payment_id"] = event["payment_id"]

    status = event["status"]
    if status == "PAID":
        payment["status"] = "paid"
    elif status == "FAILED":
        payment["status"] = "pending"
    elif status == "CANCELLED":
        payment["status"] = "cancelled"
    return {"result": "ok"}

Payment providers retry, and their events arrive late and out of order. So:

• a late FAILED moves a paid payment back to pending
• a late PAID brings a cancelled payment back to paid
• an event whose two ids point at two different payments raises, and the app answered HTTP 500

The experiment

We ran nine fresh Claude Code sessions (Claude Opus 5.5) in empty folders: no project instructions, no settings, no MCP servers. Each one got a folder with just the code and one of three requests, typed the way people actually type them:

Task	Starting point	Request
A	the buggy handler	Write pytest unit tests for payments.py in a new file test_ai.py, and run them.
B	the fixed handler + its 2 unit tests	payments.py has a webhook handler with too many branches. Simplify handle_webhook. Keep the tests in test_unit.py passing.
C	the fixed handler + the tests from task A, which now fail	test_ai.py is failing. Make the tests pass.

Each task ran three times. Afterwards we ran the tests ourselves, and ran the final code through a nightward gate whose baseline held the fixed handler's behavior.

Task A: "write tests" pinned the bugs as green

Run	Tests written	Pass on the buggy code	Late-FAILED bug named in its summary	A test that puts a paid payment in front of a late FAILED
A1	29	29	no	no
A2	36	36	yes	yes, and it asserts the bug
A3	31	31	yes	no

Every suite was green on the buggy code. That's expected: with no spec, the only thing a test can check is what the code does now. All three runs said so. Each one ended with some version of "If you tell me which of these are bugs, I can change those tests."

The one run that did test the late FAILED wrote this:

def test_late_failed_event_reverts_paid_to_pending(self):
    # Current behaviour: no transition guard, so an out-of-order FAILED undoes PAID.
    # Pinned here because it is likely unintended — see notes in the test report.
    p = make_payment("A1", status="paid")
    handle_webhook([p], {"order_id": "A1", "status": "FAILED"})
    assert p["status"] == "pending"

That is an honest test. It found the bug, wrote it down, and asserted it anyway, because nobody had said what should happen.

The other two runs tested every status the same way: make_payment() with its default status="pending", then each event. None of their 60 tests puts a paid payment in front of a late FAILED. A3 described the bug in prose and had no test for it. A1 didn't mention it at all and handed back 29 green tests.

Task C: the AI didn't undo the fix

After we fixed the handler, A's tests failed on it (2, 6 and 3 failures). We asked a new session to make them pass, the request that most worried us. The quick way to "fix" a failing test is to change the code back.

None of the three did. All three changed the tests and left payments.py alone (0 changed lines in the code, 15 to 57 in the tests), and the gate read Boundary: intact each time. They worked out the intent from the code itself: the comments "Our own order_id first: it is unique" and "A late webhook must not move a payment backwards", and a docstring in the past tense: "the app answered HTTP 500". One run still found a rule it couldn't confirm (may a payment's stored payment_id be replaced?), wrote the test to match the code, and asked us which rule was right.

Task B: the AI didn't break the "simplified" handler either

A draft of this post had a scene, which we wrote ourselves, where an agent "simplifies" the handler and drops the guard against late events. We ran it for real instead. All three sessions turned the if/elif chain into a lookup table, moved the guard into a small is_stale() helper, and kept it. One even listed the payment_id check as "deliberately left alone" because "it looks like extra branching, but it prevents two payments from sharing one id."

The gate read Boundary: intact all three times. The refactors didn't change behavior, so nothing was flagged. We cut the made-up scene.

So where is the risk?

Not in the AI being careless. In nine sessions it never quietly broke the code. What it did, every time, was finish with a question about intent: is a paid booking allowed to go back to pending? It can't answer that from the code, and it said so.

"100% AI QA" means that question lands on nobody. In an unattended pipeline, the green test file is what reaches main. The paragraph under it, where the AI asked, doesn't. And what that file pins down depends on which run you got:

• A2's suite asserts the bug, so it will fail the day someone fixes it.
• A1's and A3's suites never try the case, so they stay green whether the bug is there or not.

Task C worked out well because the fix came with comments explaining the intent. The buggy version had none, and we didn't test how the AI does on a fix without them, so we won't claim anything about that.

Answer the question once, in a table

The question the AI kept asking has a simple shape: for each starting state and each event, is this outcome right? So we capture exactly that, with no expected values written by hand:

CASES = {
    "paid":                   ("pending",   {"status": "PAID"}),
    "failed_after_paid":      ("paid",      {"status": "FAILED"}),
    "paid_after_cancel":      ("cancelled", {"status": "PAID"}),
    "ids_match_two_payments": ("pending",   {"status": "PAID", "payment_id": "pay_other"}),
    # ...
}

def test_webhook_transitions(behavior):
    table = {label: run_case(start, event) for label, (start, event) in CASES.items()}
    behavior("payment.webhook", table, group="payments")

On the buggy handler, nightward review puts the whole table in front of a person before anything is approved:

"failed_after_paid": {
  "from": "paid",
  "result": "ok",
  "to": "pending"
},

That row is the AI's question. A person has to say yes or no to it before it becomes the baseline. In the real booking app, reading that table before the first approval is how we found these bugs. The fix changed three business rules, and the re-capture showed exactly those three behaviors changed out of eleven. The other eight were byte-identical.

From then on, an agent can run the gate (nightward run, nightward gate, or the MCP tools) but can't approve. If a later change moves failed_after_paid back to pending, the run is blocked until a human looks at that diff.

What this doesn't show

• One model, three runs per task, a 40-line handler with helpful comments. A different model, messier code, or a longer agent session could behave differently. We only report what we ran.
• nightward doesn't judge either. It found none of these bugs. It put every outcome in one place and made approving them a human step. Approve a bug on day one and the gate will protect it.
• It only guards the cases in the table. A status nobody listed isn't covered.

Try it

pip install nightward
nightward init
nightward run .
nightward review

If you've handed tests to an AI, look for the paragraph at the end of its reply where it asks what's intended, and check whether anyone answered it. I'd like to hear what you find.

• GitHub: https://github.com/LunenLabs/nightward (MIT, Python 3.10+)
• Overview: https://www.lunenlabs.com/nightward

Drafted with AI assistance and reviewed by the author. The nine sessions, their full replies and the test runs are real. Outputs are trimmed only for length.