In this post we'll examine the modern test suite through the lens of dustbowl empiricism - the practice of blindly gathering massive amounts of data to find surface-level correlation. We'll see how the term translates to issues plaguing software testing, and examine suggestions for mitigating the one that can be applied to the latter. By the end we will have connected everything through a nuanced application of a Popperian approach, with a dash of a Baconian-flavored foundation to experimental science.
Hopefully you'll find this as interesting to read as I found it to be interesting to write.
A little piece of historical prologue
There's a long standing debate in philosophy of science whether there is such a thing as a "pure fact", or are all facts always "theory laden". Following the years of the U.S. depression, psychologists at Minnesota came close to one side of that spectrum - they built a personality test that seemed to require no theory of personality, and strongly resisted shaping into one.
Their mode of operation was recording hundreds of criteria and checking them against a group of diagnosed patients and a control group of what were deemed to be "ordinary Minnesotans". Whichever criteria seemed to differentiate between the groups was kept, without any attempt at a narrative, justification or rationale. The data was just given a voice, in a manner that somewhat resembled the pre-modern way of doing science (think of Tycho Brahe endlessly recording celestial movements and trying to extract the patterns out of that, before Kepler came with an actual theory).
The approach got a nickname: dustbowl empiricism. Partly it's geography and period, but the name stuck as a joke about dryness. To our machine-learning and LLM-training accustomed eyes this kind of blind data gathering grind and blackbox mentality seems normal. If you had any instinctive knee-jerk objection it was probably about overfitting, not the overall approach of blindly gathering data and finding patterns within it. But for decades it was heavily contested for its lack of explanatory power and break from any theoretical underpinnings1. Either way, this was research that promoted data collection instead of proposing mechanisms or structures, valued the volume of findings over meaning and analysis, and if something predicted reliably, kept using it without trying to uncover what it was "really" measuring.
Cool story bro, why are you telling me this, you ask?
Well, outside of loving the sound of my own voice,
Our test suites are drifting towards dustbowl empiricism, kicking a lot of dust into the air. Mostly, it's just blocking out the light.
The modern test suite
Be it manual or automated, you know the shape of the modern test suite. Your coverage sits comfortably at a weirdly specific 93.46%, CI runs take 20 minutes (manual test cycles takes 3 weeks) and, in three years, you've caught roughly two real bugs, both of which would have surfaced in staging anyway. Having contributed what probably amounts to tens of thousands of tests and assertions to such test suites over my career I certainly know their shape. The embodiment of dustbowl empiricism, gathering lots of data, without any2 underlying narrative, theory, rhyme or reason. What those thousands of tests overwhelmingly assert is that the code does what the code does. It's an exercise in tautology testing.
What's that? It's no longer an issue because AI now implements (or god forbid, plans) your test suite? Oh, for sure, dousing this dumpster fire of a problem with gasoline seems like a sensible plan. Yes, AI-generated tests did not create this pathology (we're perfectly capable of manufacturing tautologies by hand), but AI usage did industrialise it. Ask a model to "increase coverage" and you will get precisely what you asked for: a dense, syntactically flawless thicket of assertions that walks every branch of the code / spec, gathering all the data and challenging close to nothing, because challenge needs a foundation / theory to be challenged, and there is none.
Now, in fairness, some of this is the obligatory bread and butter of regulated / big-corporate software testing and validation. Sometimes you do need to generate evidence that every main spec path was walked - that's just the way the game is played. Just don't let yourself be fooled about what it is that you're actually doing, and don't eagerly volunteer into doing it more than needed. Gathering evidence for an audit is important; Necessary, even. What it's not is an effective testing strategy. The returns diminish VERY quickly, and each new test / validation contributes only a tiny sliver of confidence overall.
The data pathology and a possible Popperian cure
In 1978 Paul Meehl (dust bowl empiricist himself, which has a dramatic irony we'll not explore here) published "Theoretical Risks and Tabular Asterisks: Sir Karl, Sir Ronald, and the Slow Progress of Soft Psychology", discussing how dustbowl empiricism blind data abundance weaponised statistical significance into a scientifically meaningless endeavour (well, at least that's one possible read of it, and the one we'll take here). The specific details regarding the psychology research of the time are less important for our specific scope, but the underlying pattern they follow is very relevant for us (those details are pretty interesting though. Maybe to be explored in future post).
Meehl's point was that theories in soft psychology (counselling, social and personality psychology) are too fuzzy to progress the way scientific knowledge is supposed to. There's nothing specific or solid enough to be refuted or corroborated, and theoretical change is mainly due to people losing interest in old theories, and chasing whatever's this month's flavour in academic circles. So huge amounts of data are being accumulated, and by their sheer volume, those pieces will always support some statistical significant cross section, but it amounts to almost nothing. No new knowledge is accumulated over time (note - knowledge as opposed to data). Just a very thin sliver of almost trivial, surface level facts.
Similarly, the huge amounts of evidence the modern test suite gathers is blindly collected into a pile of data. Yes, there are some useful indicators in that pile, but without a good theoretical foundation informing our data collection, organisation and analysis, we prove almost nothing about the state of our system. Increasing coverage results in more inner tautologies, and less previously unknown knowledge or facts.
Meehl's article title mentions two Sirs - Fisher, who gave us the test of statistical significance, and Popper, who gave us the notion a theory should be under constant refutation attempts. Meehl thought these were badly integrated and possibly incompatible, but that's mostly relevant for his criticism on the psychology research of the time. For us the takeaway is mainly what Meehl prescribes as a cure - a nuanced Popperian point of view of testing and verification.
Meehl doesn't say "be Popperian", as in just generally constantly try to refute you theories. That's too broad and doesn't entail anything actionable for soft theories in psychology or a general handwaving theory as "The system works as expected" in testing. Meehl calls for consistency tests. Wherever possible, obtain two or more non-redundant estimates of the same theoretical quantity and check whether they agree. What's that, your theory isn't specific enough to spell out its terms in a manner that would support that? My reading of Meehl is that you've just encountered an actionable, strong indicator of exactly how you should further develop your theory and "nail it to the wall" to produce these details and test / verify them.
What this means (again, by my reading of Meehl) is that you're somewhat forcing yourself into a cycle of making detailed theoretical claims about the world, and immediately testing them. Whatever the result of the test is, you will have gained real knowledge through it, not just line items to your blind data pile. That knowledge will inform, detail and mutate your theories into covering more ground (real ground, not new tautological descriptions on ground you've already covered), which would translate into more "friction area" with reality, which invites further verification, and so on and so on.
For testers, this means pinning down our fuzzy, almost non-refutable handwaving statement that "the system works" into a theory-laden cashing out of what "works" looks like in each area and strata of the system under test. This necessitates a grey-box thinking even for black-box systems. This grey-box theoretical detailing of the system state is exactly what allows boundary value testing, meaningful load testing, execution paths testing, equivalency groups testing, and all other non-trivial detailings of "the system state". All of those are great, but to Meehl's point, they should feed into consistency verifications - metamorphic testing, finding different oracles for a given state and diff-ing them relentlessly, etc.
This usually rests only on a general theory of software design and a pretty rudimentary understanding of code, and of how humans3 use it. Naturally, you're already doing these things to some extent (maybe you call it "a tester's gut"). Making the process explicit and grounded in theoretical rationale can help us keep a watchful eye on the constant drift towards gathering evidence that relate only to the fuzzy handwaving (useless) theory. That evidence is easily gathered, but won't help us refute or corroborate anything concrete. This is data, not knowledge.
So a Popperian approach is our North Star, and we're gesturing at a broad outline we can follow. Is there anything more concrete we can say? Let's shed some Baconian light on the matter.
Bacon and the tests that actually decide
Centuries before Meehl and Popper (1620), Francis Bacon formalised the modern pillars of methodological experimental science in his Novum Organum (well, he boldly proposed concepts, in retrospect, they fit very nicely as the foundations). Part of that was drawing a distinction between two classes of experiments.
There are experimenta fructifera - fruit-bearing experiments - done for the immediate practical yield where the data gathering grind is the be all and end all. While those compose the bulk of scientific work, they are only part of the picture. And there are experimenta lucifera, light-bearing experiments, dealing with causes and axioms rather than immediate utility. These form the foundation that later lets you interpret data against the backdrop of your theory. Bacon's argument was that we drift towards over-investing in experimenta fructifera and somewhat starve experimenta lucifera. Or in a more nuanced reading of his body of work, we often seek "fruit" prematurely, before the axioms exist to generate it. Again the point4 isn't that data gathering is wrong, but that you need the guidance of a detailed theory to turn data into knowledge.
Bacon noted the instantia crucis, the instance of the fingerpost, named for the fingerpost planted where a road forks to different paths (or its more readable term - decisive instance). When our theory has been detailed enough through experimenta lucifera, we will be able to locate places where evidence has the potential to disagree with a one branch in the theory and corroborate another. A good test has the ability to produce exactly this kind of evidence, and it's a sharper formulation than coverage will ever manage. This mixes some of Bacon's concepts into a more unified construct than he originally sets out, but I believe that translates better to our scoped borrowing of his ideas.
From this very scoped point of view, Popper may snap into focus as a special case of Bacon. Bacon's crucial instance is a fork between any two branches of a theory; falsification is the particular fork whose branches are the theory holds and the theory is dead (well, Quine would have something to say about that, but that's for another day). So falsification becomes the ruling out of a certain path; and attacking or refuting our theory in general becomes cashed out into locating these crossroads and formulating a decision test based on the light experimenta lucifera shed on what "the state of our system" means.
Epilogue
At the end of the day it all circles back into the worth of a test being not in a mere extension of coverage, but in removing ambiguity. The formulating of such tests, and interpreting their results in a way that actually accomplish that mandates a theory detailed enough to transform our data into knowledge.
Which gives you one actionable review question for your test plan or for the 400 cases an agent just generated for you. Not "does this pass" or "what does this cover". Rather, which two states of the system does this tell apart? If there's a fork, you're casting light. If there isn't, you're kicking up dust.
To make things a bit more concrete, here's the obligatory mickey-mouse example. Take the canonical tautology: expect(add(2, 2)).toBe(4). This tells nothing apart. A world where add is correct and a world where add returns a hardcoded 4 produce identical green ticks; the test sits at no fork. Now assert a property instead: for any a and b, add(a, b) === add(b, a), and add(a, 0) === a. Suddenly there's a fingerpost. The commutativity check partitions "real addition" from "concatenation that happens to look right on 2 + 2"; the identity check kills the hardcoded-constant world outright; and the variable hooks allow feeding your favorite boundary, equivalence and anomalous value generators into your consistency checks.
If this sounds vaguely like what you've been implicitly doing your whole career, that tracks. The English "Crucial" traces back exactly to that part of Bacon's writing, talking about instantia crucis, the cross-shaped post where the road divides. If you've ever referred to something as the crucial test, you've actually been claiming, with 400 years of authority, that it marks a fork regarding the system state.
In any case, none of this is a problem you solve for once. Coverage is measurable and theory-building isn't, and human organizations are KPI optimizing by nature. It's natural for our test suite to drift into the data gathering grind.
Just make sure enough light peers through the dust you're kicking up.
-
(on a personal note from a layman's POV - given the lack of consistency and evidential justification for some of the psychological theories of the time, a radical break from the theoretical basis wasn't necessarily a bad thing). ↩
-
Of course there is an underlying narrative, theory, rhyme and reason. We're all professionals, it doesn't need to be vulgar, or thoughtless or cynical, but somehow this is where we usually end up at. Structuralists and economists would have a field day with how modern KPI weaponisation has corrupted the modern test suite 🤷♂️. ↩
-
What's this? A teaser to how things break when it's not a human writing code in a human-way-of-work manner?. ↩
-
At least the point as my biased reading according to my pre-decided narrative of the claim I'm making. ↩
Top comments (0)