Every code review I have ever sat in, sooner or later someone says it:
Fixing a bug in production costs 100 times more than fixing it in design. IBM found that.
It is the load-bearing number behind "shift left", behind "add tests", behind moving review earlier and paying for it now. So I went looking for the study.
What I found is more interesting than a debunk, because the number is not fake. It is real, it comes from somewhere specific, it describes a specific kind of software, and it gets sold as universal. Here are the four checks I run on any number someone quotes at me.
1. Who counted?
"IBM found 100x" merges two different lineages, and neither is a study of your codebase.
The IBM Systems Sciences Institute citation traces back to a 1981 set of course notes, "Implementing Software Inspections". The Institute was real as an organisation: an internal IBM training unit in Los Angeles, on Wilshire Boulevard. But a training unit is not a research programme. The cleanest trail I found is a write-up tracing that citation (gist); it describes the evidence that the institute existed at all as "scant", and its famous 1 : 6.5 : 15 : 60-100 ratios as notes, not measurements.
The other lineage is data. Barry Boehm's 1981 Software Engineering Economics contains a cost-to-fix curve built from a handful of named sources: Fagan's 1976 IBM-SSD work, Daly's 1977 GTE work, several TRW projects, and the SAFEGUARD project (Stephenson, 1976).
So check one narrows it: the counters are a small set of 1970s defence and telecom programmes, plus one set of course notes that keeps getting the credit.
2. Is there a measurement underneath, or only a sentence?
Here there is one. Boehm's curve exists, with a figure and a page number. This is not the shape where a claim arrives as a number but nobody ever collected the data.
That distinction matters. It means I am not saying the effect was invented. I am saying something narrower and more annoying: the number's population is doing all the work.
3. Is the sentence bigger than the number?
Boehm's curve runs roughly 1 → 5 → 10 → 50 from requirements through system test, climbing toward 100 only at operations and maintenance. The later, more famous line — "often 100 times more expensive... during the requirements and design phase" — takes the top of that range and drops the qualifier that the top belongs to delivery.
Line up the replications (there is a good normalised table in the literature; NASA's 2010 compilation has another) and you get a range, not a constant: means around 1 : 7 : 26 : 177, medians around 1 : 5 : 10 : 50. "100x" is the round number at the end of the range, repeated so often it reads as the centre.
4. What population?
This is the part a working developer should care about.
The evidence is large, formal, waterfall-era systems: aerospace, defence, telecom, 1970s. Boehm's own data shows the curve is flatter for small projects — closer to 5:1 than 100:1. Langenfeld (2016) reports a much shallower 1 : 1.6 : 4.9 : 6.7, and Hamill's survey medians sit near 1 : 3 : 27.
Then the biggest test so far went further. Menzies, Nichols, Shull and Layman (arXiv 1609.04886, 2016) looked at 171 software projects run between 2006 and 2014 — the largest study of this effect yet published — and found no evidence for a consistent delay penalty. Their conclusion: the delayed-issue effect is "not some constant across all projects", and may be "an historical relic that occurs intermittently only in certain kinds of projects". They also note the raw data behind the older conclusions was no longer reachable, so earlier claims could not be re-confirmed.
The verdict: signal, not proof
Late bugs probably do cost more, at least sometimes, in some systems. That direction is close to common sense, and some newer data supports a weaker version of it.
What does not hold up is the sentence as it is usually spoken: a precise, universal 100x, from an IBM study, applied to whatever codebase happens to be in the room. The number is real for a 1970s population. The authority is course notes. The universal claim was tested at scale and did not show up.
The version I keep: use the direction, don't defend the decimal. "Find it earlier" is cheap advice and usually right. "100x, everywhere" is a number borrowed from someone else's project, wearing a source that is not what people think it is.
I run these four checks on any number someone quotes at me — who counted, is there a measurement, is the sentence bigger than the number, what population. More worked examples in my previous post. If you have a number you can't check, put it in the comments and I'll run it line by line.
Top comments (0)