Every piece of software your business runs on is built on dozens of parts written by other people. Now and then those parts need an update, usually to close a security hole. The program restarts, the pages open, nobody sees an error. And sometimes a rule that used to work has stopped working. Without telling anyone.
It has happened to me more than once, on my own projects. Better that an automated test notices first, before an angry customer does. Here is what that means in plain words, and what to ask whoever builds your software.
Why you can't stop updating
A management system, a website with a customer area or a custom app is not only your supplier's code. It uses ready-made components: the one that reads uploaded files, the one that sends emails, the one that talks to the database. Think of them as spare parts in a car. The mechanic doesn't build the brakes, he fits them.
When a hole is found in one of those parts, a fixed version comes out. Staying on the old one means leaving open a door everybody now knows about. So you update, and that's fine. The question is what happens next.
How an update changes the rules without an error
Take a made-up but realistic example. An online shop gives a 10% discount only to newsletter subscribers. The rule lives in a setting of an external component, the one that handles the cart.
A security update comes out and the supplier installs it. In the new version that setting has a different name. The shop's code still uses the old name, and the component simply ignores it. No error message, no blank page. The shop keeps selling.
Except now everyone gets the discount. Or nobody does, and subscribers start writing to support. In the first case you find out when you look at the margins at the end of the month. In the second, when someone has already complained.
In an app that handles photos or health data, the rule that stops working can be a data protection rule. Then it is no longer about margins. It is about what you promised your customers.
What an automated test is, without jargon
An automated test is a check written once and repeated by the computer every time something changes. For the discount it would read roughly like this: "a customer who isn't subscribed puts a 100 euro product in the cart, the total must be 100; a subscribed customer does the same, the total must be 90".
It takes a few seconds to run. The supplier runs it before every update. If the total is wrong, the test turns red and the update doesn't go live until someone understands why.
That is where the value is. The test checks behaviour, meaning what the customer sees, and not the code. The code can look exactly as before and do something else.
The test that always says yes
There is a little-known trap. A test can stop checking and still pass. Picture a check that must confirm every private page asks for a password. After an update, the check no longer finds any page to look at. Zero pages checked, zero problems found: green.
That is why a good supplier also checks the opposite. They remove the rule on purpose and watch the test turn red. A test you have never seen fail is a test you can't trust.
What to ask your supplier
You don't need to code to ask these questions. They tell you whether the software you pay for is kept under control or whether everyone is hoping for the best.
- Are there automated tests on the rules that involve money or data? Prices, discounts, permissions, deadlines, privacy. Start with whatever is expensive when it breaks.
- Who watches the external components, and how often? There are tools that flag known holes in the parts you use. Ask whether they run and who reads the warnings.
- What happens after an update? The right answer is "the tests run, and if one is red we don't release". The answer to dig into is "we click around a bit by hand".
- Has a test ever stopped an update? If yes, that's a good sign: the tests are actually checking something.
- If you change supplier, do the tests stay? They are part of the project you pay for. They belong in your repository, next to the code.
What it costs
Writing tests takes time, so money. You don't do it for everything. In an average management system the truly delicate rules are a few dozen. One test each takes a few hours, and then it works for free for years.
The other side of the sum is what a wrong discount costs you for a month, a customer who can't log in, or a privacy promise that is no longer kept while nobody knows.
For the technical reader
Both traps share a shape: the line of code is still there, and it no longer does anything. Nothing below is tied to one framework; I have seen it in Python, and the same holds in PHP or JavaScript.
A setting under a name nobody reads
Many libraries are configured by assigning a class attribute at startup. If a new version renames that attribute, Python does not complain about the old name: assigning an attribute that does not exist on a class simply creates it.
The app starts, every test that only checks "does it run" stays green, and the library quietly falls back to its default. The defence is a test that reads the value back under the current name and fails loudly if the name moves again.
It looks redundant next to a behavioural test. It is not: when it fails it tells you exactly what broke, and if the attribute is renamed once more, the AttributeError shows up in CI instead of in production. Keep a behavioural test as well, because the setting is only a means.
A loop over an empty list
Tests that walk every route, file or table are handy: nobody has to remember the route added next week. But a framework release can change how mounted routers appear in the route list. After that, the filter finds zero items, the loop runs zero times, and the test is green.
One check that the list is not empty, placed before the loop, turns a silent no-op into a red test.
Habits that catch both
- Run the dependency audit inside the test suite, so the warning lands next to the results and the upgrade starts from there.
- Before trusting a guard test, remove the guard and watch the test fail. A test you have never seen fail may be testing nothing.
- After a big version jump, read the release notes for renames, but don't rely on them: no changelog will tell you that your own test now loops over an empty list.
I have also written about security updates in the post on two critical PHP flaws in October, with the steps I followed on my own services. If you want to know how well the software you use is protected, or have one built with the right tests from day one, we can talk it through on a free first call.
Originally published at gabrielepieretti.dev. I write about software for small businesses, Laravel, FastAPI and privacy-by-design. More here.
Top comments (0)