DEV Community

SDET Code
SDET Code

Posted on

The AI fixed the bug. Every test passed. The customer lost free shipping.

#python #pytest #testing #ai

Here is a pet-shop checkout. It is small on purpose, but the coupling in it is the kind every real codebase has: two features quietly depend on a third.

TAX_RATE = 0.10        # 10% tax
LOYALTY_PER = 10.0     # 1 loyalty point per $10 of order total
FREE_SHIP_AT = 30.0    # free shipping (and silver tier) at/above this total


def checkout_total(items):
    subtotal = round(sum(i["price"] * i["qty"] for i in items), 2)
    tax = round(subtotal * TAX_RATE, 2)
    return {"subtotal": subtotal, "tax": tax, "total": round(subtotal + tax, 2)}


def loyalty_points(items):
    total = checkout_total(items)["total"]            # derived from total
    return {
        "points": int(total // LOYALTY_PER),
        "tier": "silver" if total >= FREE_SHIP_AT else "bronze",
    }


def shipping_fee(items):
    total = checkout_total(items)["total"]            # derived from total
    free = total >= FREE_SHIP_AT
    return {"fee": 0.0 if free else 3.5, "free_shipping": free}
Enter fullscreen mode Exit fullscreen mode

A customer has three bags of dog food and a leash in the cart: $27.30 before tax, $30.03 after. Just over the $30 line, so they get free shipping and silver tier.

The fix

Now ask an AI agent to make the tax calculation "more precise". It changes one line:

-TAX_RATE = 0.10
+TAX_RATE = 0.095
Enter fullscreen mode Exit fullscreen mode

The tests run:

$ pytest -q
...                                                    [100%]
3 passed in 0.01s
Enter fullscreen mode Exit fullscreen mode

Green. The agent reports that the task is done.

But the total is now $29.89. The customer dropped below $30, so they lost free shipping and silver tier. Nothing failed, because no test said "this cart ships free". Nobody writes that test. You test the thing you are changing, not the two features that happen to read its output.

That is the pattern with AI-made changes: the agent fixes A, and B, C and D move with it. The suite stays green because nothing pins down B, C or D.

What it looks like with a gate

We ran the same change through nightward, a pytest plugin we built for exactly this. The output below is from the actual run, trimmed only for length:

$ nightward run .
3 passed in 0.02s

Boundary: breached (3 unapproved)
unchanged=0 changed=3 new=0 removed=0

group: billing
  - [CHANGED] checkout_total

group: fulfillment
  - [CHANGED] shipping_fee

group: loyalty
  - [CHANGED] loyalty_points
Enter fullscreen mode Exit fullscreen mode

One line changed, three behaviors moved, in three different parts of the product. nightward review shows exactly what moved:

[CHANGED] checkout_total
-  "tax": 2.73,
-  "total": 30.03
+  "tax": 2.59,
+  "total": 29.89

[CHANGED] shipping_fee
-  "fee": 0.0,
-  "free_shipping": true
+  "fee": 3.5,
+  "free_shipping": false

[CHANGED] loyalty_points
-  "points": 3,
-  "tier": "silver"
+  "points": 2,
+  "tier": "bronze"
Enter fullscreen mode Exit fullscreen mode

And the gate stays closed until someone decides:

$ nightward gate
boundary breached (3 unapproved)
$ echo $?
1
Enter fullscreen mode Exit fullscreen mode

15-second demo: an AI fixes one line, every test passes, nightward shows three customer-facing behaviors moved

No expected values

Here is the whole test file. Notice what is missing:

from shop import checkout_total, loyalty_points, shipping_fee

CART = [
    {"sku": "dog-food", "price": 8.50, "qty": 3},
    {"sku": "leash", "price": 1.80, "qty": 1},
]


def test_checkout_total(behavior):
    behavior("checkout_total", checkout_total(CART), group="billing")


def test_loyalty_points(behavior):
    behavior("loyalty_points", loyalty_points(CART), group="loyalty")


def test_shipping_fee(behavior):
    behavior("shipping_fee", shipping_fee(CART), group="fulfillment")
Enter fullscreen mode Exit fullscreen mode

There are no assert lines. The behavior fixture records what the code returns today. The first time you run it, a human reads those values (nightward review) and approves them once (nightward approve). They are saved as plain JSON next to your code and committed:

{
  "env": {
    "os": "Windows",
    "python": "3.12"
  },
  "group": "loyalty",
  "name": "loyalty_points",
  "payload": {
    "points": 3,
    "tier": "silver"
  },
  "source": "test_shop.py::test_loyalty_points"
}
Enter fullscreen mode Exit fullscreen mode

From then on, every run compares against that approved baseline and lists everything that moved, grouped by feature.

If you have ever put off testing because writing the expected values felt like the real work, this is the part that changes. You don't decide what the right answer is up front. You look at what the code does now and say "yes, that".

What it does not do

This matters, so it gets its own section.

  • It doesn't know what is correct. nightward cannot tell you that bronze tier is a bug. Maybe the new tax rate is the law and the customer really should pay shipping. It only makes sure the change can't pass silently. A human either approves it (intended) or fixes the code (regression).
  • It only guards what you capture. If no test captures shipping, a shipping change goes unnoticed. Capture the outputs your users would notice.
  • It is not a test generator. It doesn't write tests for you. It turns the tests you have into a gate.

Timestamps, UUIDs and similar volatile values are normalized before comparison, so a gate that fires on every run is treated as a bug. A noisy gate is a dead gate.

Built for AI agent loops

The split is deliberate: an agent can run the gate and read the verdict (nightward run, nightward gate, or the nightward_run / nightward_status MCP tools), but approving is a human-only CLI step whose result is a file diff that goes through code review. The agent gets a definition of "done" that it can check but not change.

Try it in five minutes 

pip install nightward
Enter fullscreen mode Exit fullscreen mode

Add a test that calls behavior(name, value, group=...) on something your users see, then:

nightward init
nightward run .
nightward review
nightward approve --all
git add .nightward && git commit -m "nightward baseline"
Enter fullscreen mode Exit fullscreen mode

After that, nightward gate in CI exits non-zero whenever an unapproved change shows up.

How do you currently check what your AI agent changed besides the thing you asked it to change? I'd like to hear what works for you, and where nightward falls short.

Drafted with AI assistance; every output above is from a real run.

Top comments (1)

Collapse
 
galagent profile image
Qingfeng •

The green suite is the interesting part here: we hit the same pattern, where an agent's change is locally correct and globally destructive, and review only catches it if someone happens to know which behaviours are load-bearing. In our experience the durable fix is not a stricter prompt but an approval boundary around the change itself, which is why we run a local-first agent runtime that gates each action, so anything that leaves the repo or cannot be replayed from a receipt needs a human yes before it lands. Approved baselines like yours are really audit receipts, and they are worth far more when they ship next to the run log than when they live only in the test file.