The Blueprint · No. 5. Start with No. 1: The Setting You Never Changed.
In the Second World War, the American military had a problem with its bombers. Too many were being shot down, and the obvious remedy was to fit armour to them. But armour is heavy, a plane can carry only so much, and cover it everywhere and it will barely leave the ground. So the real question was where. Which parts of the plane most needed the protection.
They had data to answer it. Returning bombers were inspected, and the engineers could see where the hits had landed. In the telling that became famous they clustered in a familiar pattern: heaviest along the fuselage and the wings, the engines coming back comparatively clean. Reinforce the parts taking all the fire, the reasoning went, and you will save the most planes. It is a hard argument to fault. You have the data, the damage is right there in front of you, and you are only following it.
A statistician named Abraham Wald looked at the same figures and reached the opposite conclusion. The armour belonged where the holes were not. Protect the engines, the clean areas, the places nobody thought to patch.
The reason, once you hear it, rearranges something in your head and does not put it back. The engineers were studying the planes that came back, because those were the only planes they had. A bomber covered in holes across its wings and fuselage was a bomber that had been hit in those places and had still flown home, which meant those were exactly the spots where a plane could take a beating and survive.
The clean areas were clean for a reason the data could not show. The planes that were hit there did not return to be inspected. They were somewhere in the sea. Those absent holes marked the wounds that killed.
That is the whole trap, and it is worth seeing plainly, because it has nothing to do with aeroplanes. Whenever you draw a lesson from a set of examples, something decided which examples reach you, and that something is rarely chance. It is a filter. Survival, success, fame, memory, simply staying in business: each of those filters does its work by removing the failures before they ever arrive.
So the set in front of you is only the residue left once the filter has run, a long way from a fair slice of everything that was tried, and because the filter's whole job was to take things away, the things it took away are the ones you cannot see. They are also, very often, the ones you most need.
You feel how strong this is the moment you start looking for it. Pick up any book about how some billionaire built their company and you will find a handful of habits offered up as the cause: the early mornings, the ferocious focus, the refusal to hear the word no. What you will never find, because nobody writes that book, is the far larger pile of people who rose at the same hour and focused just as fiercely and refused just as hard, and went broke regardless.
If the people who failed had every one of the winning habits too, the habits cannot be the thing that set the two apart. You are being handed the survivors and asked to reverse a recipe from them, with the one ingredient that could tell you what actually mattered, the failures, quietly deleted from the page.
It runs through much smaller things as well. When someone tells you they do not make things like they used to, waving at a hundred-year-old chair that is still rock solid, remember that you are looking at the one chair that lasted a century. Most of the flimsy furniture of the past broke and was thrown out generations ago. You are holding the best of the old, the sliver that survived, up against the everyday run of the new.
So here is the move, and it is a single question you can put to almost any claim built on examples. What decided which cases I get to see, and what would the ones it left out have looked like?
Before you copy the habits of the successful, go looking, in your imagination if nowhere else, for the people who did the very same and failed, and ask whether they shared the habit. Before you decide the old ways were better, ask what broke and disappeared long before you arrived to inspect what was left. The absence is shaped, and its shape is the part of the answer that did not survive to be seen.
None of this means every set of examples is lying to you. Sometimes your sample really is a fair one, gathered without a filter quietly picking the winners, and then this particular problem does not arise. The trap springs only when the very process that produced your evidence is the same process that removed the counter-examples. The tell is easy to learn once you have it. Your data is made of survivors, of winners, of the things that lasted and the stories that got told. The moment you notice that, you know to go looking for the planes that did not come back.
Note: Wald's wartime analysis survives as eight memoranda for Columbia University's Statistical Research Group; the standard scholarly reconstruction is Marc Mangel and Francisco Samaniego, Abraham Wald's Work on Aircraft Survivability, Journal of the American Statistical Association 79 (1984): 259–267. The memoranda estimate the vulnerability of each part from the damage on returning aircraft; the familiar bullet-hole diagram and the scene of engineers overruled in a briefing room are later retellings, not Wald's own.
From The Blueprint, a series on The Durability Curve about the surfaces hidden inside systems you already live in. If this changed how you read a set of examples, subscribe.
Top comments (0)