On a Friday morning our carrier partner's account manager rang to ask why we had created six thousand four hundred and twelve shipments in their production system over three nights, all of them for a customer called Test Tester, most of them addressed to a street that does not exist.
Our integration suite runs against a stubbed partner. A WireMock container comes up beside the tests, the suite points at it, and the whole thing takes four minutes. That had been true for two years. In August we upgraded our internal client library to its 2.x line, which renamed the setting it reads for the endpoint and, at the same time, gave that setting a compiled in default of the partner's live URL, because that is what production uses and a default that works in production looks like a kindness to whoever writes it.
Our test configuration still set the old name. The client saw no value, used its default, and called the real system with the real credentials, which were sitting in the job environment because the nightly smoke test runs in the same workflow. The stub came up, received nothing at all, and shut down. The suite passed. That is the part that took me longest to accept: the tests were green because the live system behaves exactly like our stub for the happy path. We had built a suite that could not tell the difference between a fixture and a real company.
Three changes. The client now refuses to construct without an explicit base URL and there is no default anywhere in it, so absent configuration is a crash rather than a decision made in 2024 by somebody else. Test jobs run with egress blocked to everything except the compose network, so a misrouted call fails with a connection refused instead of succeeding somewhere real. And the stub asserts at teardown that it received the calls the test claimed to make, which turned up two more suites that had been quietly talking to nobody.
A green suite tells you nothing threw. It cannot tell you what it was talking to, and mine could not have told me if it tried. The only boundary you can rely on is the one your tests are physically unable to cross.
– Sergey Shinder
Top comments (0)