Our shared parent POM moved Surefire from 2.22.2 to 3.2.5 on a Tuesday, as part of a routine platform bump that touched nineteen repositories. Two of them went red immediately. Thirty seven tests, all in the billing and tax modules, all failing on assertions about values none of them had written.
We spent three days treating this as a Surefire problem, because that is what had changed. We pinned the old version and the tests went green, which felt like a diagnosis and was actually just the symptom moving. What the upgrade had changed was the default order in which test classes run. Nothing else. The same tests, the same code, a different sequence.
A colleague reproduced it locally by running the suite reversed, and the cause came out in an afternoon. One integration test class inserted a tax rate row into the Testcontainers Postgres at setup and never removed it, on the reasonable grounds that the container is thrown away afterwards. Nineteen tests in two other classes read that row and had never created one. They had passed for two years because the class that wrote it happened to sort earlier in the file system. A second group failed for a related reason: one test set the JVM default time zone to Europe/London and left it set, and four date boundary tests were quietly relying on it rather than on their own fixture.
What changed is small and has held. Every test class now owns its data, created in its own setup and rolled back after, with a shared base class that makes that the path of least effort. The suite runs in a random order with the seed printed at the top of the log, so a failure can be reproduced exactly. The nightly build runs it twice with two different seeds, which found five more couplings in the first fortnight. And nothing may set a JVM global without restoring it, enforced by a rule in the build rather than by memory.
A suite that only passes in one order is testing a sequence, not the code. Ours had an agreement with itself that nobody had written down, and the first thing to disturb it got three days of blame for a fault we had shipped ourselves.
– Sergey Shinder
Top comments (0)