DEV Community

Cover image for Did the upgrade break the tests, or were they already failing?
Versyka
Versyka

Posted on AI-assisted

Did the upgrade break the tests, or were they already failing?

#testing #programming #python #devops

Most real repositories have a few tests that already fail. A flaky one, a test that needs a service nobody runs locally, one that broke last month and has a ticket.

That's a problem for any automated change, because "the tests fail" stops meaning anything. Did the upgrade break them, or were they already red?

The answer is cheap: run the tests before the change, run them after, and compare.

The comparison 

before:  t1 ✓  t2 ✓  t3 ✗  t4 ✓  t5 ✓  t6 ✗  t7 ✓  t8 ✓
after:   t1 ✓  t2 ✓  t3 ✗  t4 ✓  t5 ✓  t6 ✗  t7 ✓  t8 ✓
Enter fullscreen mode Exit fullscreen mode

Same two failures. Nothing new. The upgrade didn't break anything, and the PR should say so: "t3 and t6 were already failing before this change."

before:  t1 ✓  t2 ✓  t3 ✗  t4 ✓ ...
after:   t1 ✓  t2 ✓  t3 ✗  t4 ✗ ...
Enter fullscreen mode Exit fullscreen mode

t4 is new. That's the upgrade, and it shouldn't become a PR yet.

Doing it with pytest 

# before
pytest -q -p no:cacheprovider --junitxml=before.xml || true
# apply the upgrade, then
pytest -q -p no:cacheprovider --junitxml=after.xml || true

python - <<'EOF'
import xml.etree.ElementTree as ET
def failed(f):
    return {f"{c.get('classname')}::{c.get('name')}" for c in ET.parse(f).iter("testcase")
            if c.find("failure") is not None or c.find("error") is not None}
b, a = failed("before.xml"), failed("after.xml")
print("already failing:", sorted(b & a))
print("NEW failures:   ", sorted(a - b))
print("fixed:          ", sorted(b - a))
EOF
Enter fullscreen mode Exit fullscreen mode

The same idea works with Jest (--json), Go (go test -json) or JUnit XML from Maven and Gradle.

What to do with a new failure

Don't open the PR. Read the error, fix the code, run again. Cap the attempts so a hopeless change doesn't loop forever, and when you give up, say why in plain words instead of shipping something red.

How Versyka uses it

This is the rule Versyka applies to every upgrade it proposes: tests run before and after in a sandbox; already-failing tests don't block the PR and are listed in it; a new failure means no PR, up to three attempts, then a note explaining what blocked it.

It's also just a good habit for humans reviewing their own upgrade branches.

Top comments (0)