Software developers are accustomed to an uncomfortable fact: passing the tests does not prove that the software is free of bugs. It proves that the software passed the tests. These are not the same proposition. One may write ten thousand unit tests, watch ten thousand little green ticks march obediently across the screen, deploy the application, congratulate oneself on the elegance of the build, and discover six minutes later that a customer in Belgium has entered an apostrophe into a field nobody expected to contain an apostrophe and brought the entire thing to its knees. This is software. We have learned to live with it.
Cybersecurity reviews suffer from a related, and rather more troublesome, problem. Suppose I install a security product on a company's systems and twelve months pass without incident. Was the product effective? Perhaps. Or perhaps nobody attempted anything particularly serious. Perhaps somebody did and another security control stopped it. Perhaps an attack succeeded but remained undiscovered. Perhaps the product quietly prevented seventeen attacks and deserves a medal. From the observable fact that nothing happened, we have somehow arrived at the conclusion that the security software worked. There is rather a lot of unexplored territory between those two statements.
That is the evidence problem in cybersecurity reviews.
The threat comes before the feature list
One of the easiest ways to review security software is to count things. Product A has malware protection, ransomware protection, phishing protection, a VPN, a password manager and something involving artificial intelligence. Product B has all of these things except the password manager. Product A therefore wins six features to five.
Simple, certainly. Useful, not necessarily.
Before evaluating a security product there is a more important question: what is it supposed to protect, and from what? A password manager, an antivirus application, an endpoint detection platform, an external attack-surface management system and a penetration-testing service can all be placed under the enormous heading of cybersecurity, but they are solving quite different problems.
Think of a building. A lock attempts to prevent somebody entering. An alarm attempts to detect somebody entering. A security inspection tries to discover how somebody could enter. A security guard responds when somebody actually does. It would be peculiar to review these things by counting their features and declaring the alarm superior to the lock because it comes with a mobile application. Yet software comparisons routinely do something rather similar.
At The Tool Money Lab, our cybersecurity research therefore begins with the threat model. Who is being protected? What is being protected? Against what sort of threat? At which stage of an attack does the product operate? What happens after it discovers something? Until those questions have answers, most feature lists are little more than decoration.
A vendor claim is evidence of a claim
Developers will recognise another important distinction. Suppose the documentation for a library says that it performs an operation in 20 milliseconds. You may accurately report that the documentation states the operation takes 20 milliseconds. You cannot necessarily conclude that the operation takes 20 milliseconds under every meaningful circumstance. For that, you would want to know something about the hardware, workload, test conditions, sample size, measurement method and what happens when the database contains rather more than twelve rows.
Cybersecurity deserves the same discipline. If a company describes its product as providing continuous penetration testing, that description is useful evidence. It is evidence of what the vendor says the product does. It is not automatically independent proof of how comprehensively, reliably or effectively the product performs that function.
This matters particularly when numbers appear. A detection rate of 99.8 per cent looks impressively authoritative. Numbers have that effect. Add a decimal place and almost anything begins to resemble physics. But before treating 99.8 per cent as an established property of the product, it is reasonable to ask who conducted the test, what was tested, against what, when, under which configuration, how large the sample was and whether the company selling the product paid for the research.
None of those questions makes the statistic false. They tell us what sort of evidence it is. That distinction matters.
Research is not hands-on testing
There is a peculiar habit in online software publishing whereby the phrase "hands-on review" sometimes appears to mean that somebody looked at the company's website rather carefully.
These are different activities.
Reading technical documentation is research. Watching a product demonstration is observation. Installing the product and using it is experience. Designing controlled scenarios to establish how it behaves is testing. None of these is worthless, but neither should one masquerade as another.
This is especially important in cybersecurity, because meaningful testing can be difficult. A reviewer can install a password manager and determine fairly quickly whether importing credentials works. Evaluating an enterprise penetration-testing platform is another matter. One needs an appropriate environment, defined targets, technical knowledge, controlled conditions, permission to conduct the activity and some idea of what the expected result ought to be.
Clicking around a dashboard for twenty minutes and announcing that the product provides excellent protection would be wonderfully efficient. It would not tell us much.
For that reason, TTML distinguishes documentation-based research from experience-informed research and genuine testing. If we have not tested something, we should not write as though we have. There is nothing wrong with saying, "According to the vendor's documentation..." Those few words tell the reader something important about the provenance of the claim.
“Not confirmed” is a useful answer
Databases dislike uncertainty. People building databases often dislike it even more. There is an empty field, and the empty field wants to be filled.
Suppose we are comparing two security products and encounter the question, "Does Product B offer a free trial?" We cannot find one. The temptation is to write "No" and move on. The table is complete and everybody is happy.
Unfortunately, we have answered a different question.
What we established was that we could not confirm that Product B offers a free trial. We did not establish that Product B does not offer one. Those statements are not equivalent.
This is why TTML uses phrases such as "Not confirmed", "Not publicly listed" and "Pending editorial review". They are not particularly glamorous, but uncertainty is itself information. When the evidence does not support either yes or no, the accurate answer is sometimes neither.
Developers ought to be comfortable with this idea. There is, after all, a reason programming languages have spent so much time arguing about null.
Compliance is not a feature you switch on
Cybersecurity websites contain an impressive collection of capital letters: SOC 2, ISO 27001, PCI DSS, HIPAA, GDPR. Arrange enough of them in a row and an otherwise ordinary SaaS homepage begins to look as though security clearance is required to enter.
The difficulty is that several quite different claims may sit behind a compliance logo. A vendor may hold a certification. Its software may provide controls useful to organisations pursuing certification. A platform may collect evidence useful during an audit. A service may support activities associated with a regulatory requirement. These things are related, but they are not identical.
In particular, buying software does not ordinarily make an organisation compliant. Compliance usually involves technology, processes, controls, people, documentation and continued organisational behaviour. Installing a product and declaring the problem solved would be rather like buying a fire extinguisher and assuming that every fire regulation affecting the building has now been satisfied.
So when reviewing security software, we try to preserve the actual meaning of the claim. "Supports compliance with" should not quietly turn into "makes you compliant with". "Designed to help satisfy controls" should not become "certified for". The difference may be only a few words, but they are important words.
Comparisons make weak evidence worse
A badly researched product review can misrepresent one product. A badly researched comparison can misrepresent two products and then manufacture a winner.
This is quite an achievement.
Comparison pages encourage certainty because their structure demands answers. Which product is cheaper? Which provides better protection? Which is easier to deploy? Which supports more compliance frameworks? Which should you buy?
A table containing seventeen instances of "Not confirmed" is admittedly less satisfying than one full of green ticks and red crosses. But the purpose of the table is not to satisfy the table. It is to help somebody make a decision.
Suppose Product A publicly documents ten capabilities and Product B publicly documents seven. It is tempting to conclude that Product A has more capabilities. Perhaps it does. What we actually know, however, is that we found more documented capabilities for Product A. Product B may lack the other three, or its documentation may be worse, or the same functions may exist under different names, or we may simply have failed to find them.
That is why comparisons need evidence on both sides. Otherwise uncertainty has an unpleasant habit of turning into a competitive disadvantage, and the vendor with the better documentation wins rather than the vendor with the better product.
A good product can still be the wrong recommendation
Eventually, research has to become useful. Readers are not studying cybersecurity products because they have developed an unusual affection for feature matrices. They want to make a decision.
This creates another temptation: recommend the product. If an article is about Product X, surely the conclusion ought to be that somebody should buy Product X.
Not necessarily.
A sophisticated continuous-security-validation platform may be excellent software and a terrible recommendation for a five-person company with no security team. An inexpensive consumer antivirus product may be perfectly sensible for a household and entirely inappropriate for a financial institution.
The question is not merely whether a product is good. The more useful question is: for whom is it good? The corresponding question matters just as much: who should look elsewhere?
A useful review should occasionally persuade a reader not to buy the thing being reviewed. Otherwise it is not really a review. It is an advertisement with unusually long paragraphs.
Affiliate relationships make evidence discipline more important
There is an obvious complication when a publication reviewing software also participates in affiliate programmes. Sometimes the publication can earn money when somebody buys the product.
Pretending that this creates no conflict would be silly. The sensible response is not to pretend the incentive does not exist, but to design the editorial system so that the incentive does not determine the evidence.
Affiliate relationships should be disclosed. Commercial links should be identifiable. The existence of a commission should not cause an undocumented product capability to materialise. The absence of a commercial relationship should not make a competitor's strengths disappear.
This is partly an ethical question, but it is also a system-design question. If commercial relationships and editorial evidence are represented as separate concerns, it becomes much harder for one to quietly rewrite the other.
Developers already have a name for this idea: separation of concerns. It turns out to be useful outside codebases too.
The evidence trail matters more than the decimal point
The longer we work on software research, the less interesting the final numerical score becomes. A score such as 9.2 out of 10 looks wonderfully precise, but what does the ".2" actually represent? Was the product measurably two-tenths better than one scoring 9.0? Could another reviewer reproduce the result? Would the figure survive a pricing change next Tuesday?
Precision and accuracy are not the same thing. Sometimes precision merely makes uncertainty look expensive.
The more useful questions concern the evidence underneath the conclusion. What did the vendor document? What did independent sources establish? What did we observe? What did we test? What remains uncertain? Who is the product designed for? What evidence would cause us to change our conclusion?
That produces fewer spectacular declarations, but it produces something much more useful: a review which can be updated when reality changes.
And reality does change. Security products change. Threats change. Prices change. Certifications change. Companies are acquired. Features disappear. Marketing departments discover artificial intelligence.
A cybersecurity review ought therefore to behave less like a monument and rather more like maintained software. Review the dependencies. Update the evidence. Remove claims which are no longer supported. Do not assume that because yesterday's build passed, tomorrow's will.
There is nothing wrong with saying “we don't know”
This may be the least fashionable conclusion available on the internet, but we do not know everything. Neither does the vendor. Neither does the reviewer. Neither, inconveniently, does the attacker.
Cybersecurity operates in an environment of adversaries, incomplete information and extraordinarily complicated systems. Absolute claims should therefore make us more suspicious rather than more comfortable.
The purpose of a cybersecurity review should not be to eliminate uncertainty by writing confidently. It should be to describe the uncertainty accurately enough that somebody can make a better decision despite it.
Developers already live with this principle. A passing test tells us something. A failed test tells us something. A test we never wrote tells us nothing at all.
Security research deserves the same distinction.
Our rule at The Tool Money Lab is deliberately simple: say what you know, say how you know it and, when you do not know, say that too.
It may not produce the most dramatic cybersecurity reviews on the internet. But dramatic is not generally what one wants from security engineering.
Reliable will do.
This essay is adapted from The Tool Money Lab's cybersecurity review methodology. The complete methodology, including how we handle pricing, compliance, comparisons and evidence classifications, is published in The Trouble With Cybersecurity Reviews: https://thetoolmoneylab.com/editorial/cybersecurity-review-methodology
Top comments (0)