This is the story of survivorship bias. This is the story of why we must take a step back and think.
During World War II, One problem the US military faced was how to reduce aircraft casualties. They researched the damage received to their planes returning from conflict. By mapping out damage they found their planes were receiving most bullet holes to the wings and tail. The engine was spared.
The US military’s conclusion was simple: the wings and tail are obviously vulnerable to receiving bullets. We need to increase armor to these areas.
But Abraham Wald, a mathematician, came to a different conclusion, His conclusion was surprising: don’t armor the wings and tail. Armor the engine.
Wald’s insight and reasoning was based on understanding what we now call survivorship bias. Bias is any factor in the research process which skews the results. Survivorship bias describes the error of looking only at subjects who’ve reached a certain point without considering the (often invisible) subjects who haven’t. In the case of the US military they were only studying the planes which had returned to base following conflict i.e. the survivors. In other words what their diagram of bullet holes actually showed was the areas their planes could sustain damage and still be able to fly.
Wald surmised that it was actually the engines which were vulnerable: if these were hit the plane and its pilot went down and didn’t return to base to be counted in the research. The military listened and armoured the engine not the wings and tail.
The US Airforce suffered over 88,000 casualties during the Second World War. Without Wald’s research this undoubtedly would have been higher. But his insight continues to this day and has become an issue in clinical research, financial markets and the people we choose to look up to.
SURVIVORSHIP BIAS IN CLINICAL RESEARCH
In 2010 in Boston, Massachusetts a trial was conducted at Harvard Medical School and Beth Israel Deaconess Medical Center (BIDMC) into improving patient survival following trauma. A major problem following trauma is if the patient develops abnormal blood clotting or coagulopathy. This hinders them in stemming any bleeding they have and increases their chances of bleeding to death. Within our blood are naturally occurring proteins called factors which act to encourage blood clotting. The team at Harvard and BIDMC investigated whether giving trauma patients one of these factors would improve survival. The study was aimed at patients who had received 4-8 blood transfusions within 12 hours of their injury. They hoped to recruit 1502 patients but abandoned the trial after recruiting only 573.
Why? Survivorship bias. The trial only included patients who survived their initial accident and then received care in the Emergency Department before going to Intensive Care with enough time passed to have been given at least 4 bags of blood. Those patients who died prior to hospital or in the Emergency Department were not included. The team concluded that due to rising standards in emergency care it was actually very difficult to find patients suitable for the trial. It was therefore pointless to continue with the research.
This research was not the only piece reporting survivorship bias in trauma research. Does this matter? Yes. Trauma is the biggest cause of death worldwide in the under 45 year-olds. About 5.8 million people die worldwide due to trauma. That’s more than the annual total of deaths due to malaria, tuberculosis and HIV/AIDS. Combined. Or, to put it another way, one third of the total number of deaths in combat during the whole of the Second World War. Every year. Anything that impedes research into trauma has to be understood. Otherwise it costs lives. But 90% of injury deaths occur in less economically developed countries. Yet we perform research in Major Trauma Units in the West. Survivorship bias again.
As our understanding of survivorship bias grows so we are realising that no area of Medicine is safe. It clouds outcomes in surgery and anti-microbial research. It touches cancer research. Cancer survival rates are usually expressed as 5 year survival; the percentage of patients alive 5 years after survival. But this doesn’t include the patients who died of something other than cancer and so may be falsely optimistic. However, Medicine is only a part of the human experience survivorship bias touches.
SURVIVORSHIP BIAS IN FINANCIAL MARKETS & OUR ROLE MODELS
Between 1950 and 1980 Mexico industrialised at an amazing rate achieving an average of 6.5% growth annually. The ‘Mexico Miracle’ was held up as an example of how to run an economy as well as encouraging investment into Latin American markets. However, since 1980 the miracle has run out and never returned. Again, looking only at the successes and not the failures can cost investors a lot of money.
Say I’m a fund manager and I approach you asking for investment. I quote an average of 1.8% growth across my funds. Sensibly you do your research and request my full portfolio:
It is common practice in the fund market to only quote active funds. Poorly performing funds, especially those with negative growth, are closed. If we only look at my active funds in this example then yes, my average growth is 1.8%. You might invest in me. If however you look at all of my portfolio then actually my average performance is -0.2% growth. You probably wouldn’t invest then.
Yet survivorship bias has a slight less tangible effect on modern life now. How often is Mark Zuckerberg held up as an example for anyone working in business? We focus on the one self-made billionaire who dropped out of education before making their fortune and not the thousands who followed the same path but failed. A single actor or sports star is used as a case study on how to succeed and we are encouraged to follow their path never mind that many who do fail. Think as well about how we look at other aspects of life. How often do we look at one car still in use after 50 years or one building still standing after centuries and say, “we don’t make them like they used to”? We overlook how many cars or buildings of a similar age have now rusted or crumbled away. All of this is the same thought process going through the minds of the US Military as they counted bullet holes in their planes.
To the victor belong the spoils but we must always remember the danger of only looking at the positive outcomes and ignoring those often invisible negatives. We must be aware of the need to see the whole picture and notice when we are not. With our appreciation of survivorship bias must also come an appreciation of Abraham Wald. A man whose simple yet profound insight shows us the value of stepping back and thinking.
Don’t look just at what you can see. Consider all the things that started on the same path but didn’t make it. Try to figure out their story, as there is as much, if not more, to be learned from failure.
Considering survivorship bias when presented with examples of success is difficult. It is not instinctive to pause, reflect, and think through what the base rate odds of success are and whether you’re looking at an outlier or the expected outcome. And yet if you don’t know the real odds, if you don’t know if what you’re looking at is an example of survivorship bias, then you’ve got a blind spot.
Whenever you read about a success story in the media, think of all the people who tried to do what that person did and failed. Of course, understanding survivorship bias isn’t an excuse for not taking action, but rather an essential tool to help you cut through the noise and understand the world. If you’re going to do something, do it fully informed.
SURVIVORSHIP BIAS IN EXTRASENSORY PERCEPTION
Back in the 1930s, Dr. Joseph Rhine set out to test whether or not extrasensory perception (ESP) really existed. To figure this out, he tested whether someone could successfully guess the order of a shuffled deck of cards.
He originally used regular playing cards but switched to specially designed Zener cards so that participants didn’t default to guessing only the playing cards they were familiar with versus genuinely guessing.
In his experiment, he asked 500 people to guess the order of these cards. Participants who guessed right were moved to the next round. They were asked to guess again, and those who guessed correctly were moved to the next round and so on.
Whoever was left at the end was deemed to have telepathic ability because they were able to guess correctly in every round.
As interesting as this experiment is, it was a case of survivor bias. The experiment, unintentionally, found the probability that someone, anyone, in the participant pool would guess correctly each time. It didn’t determine that a particular person had abilities.
When it comes to your product campaigns, ask yourself why one campaign performed better than others. While it might seem as though you’ve found the secret sauce to success, the success might be attributed to luck of the draw. Out of everyone in your industry, you’re the lucky one who got it right.
Not to sound too negative, but in some cases it has more to do with luck than things you learned along the way.
SURVIVORSHIP BIAS ENTREPRENEURSHIP
You’re in business to succeed, and you’ve probably read your fair share of blog posts about entrepreneurs who’ve done well.
So what’s the problem here? Anyone thinking about becoming an entrepreneur has looked at these mega-millionaires and tried to discern a pattern to their success. Some invested all their money while others— like Mark Zuckerberg, Jack Dorsey, and Travis Kalanick— dropped out of college to pursue their dream.
As simple as those options might seem, following those paths ignore the fact that for as many people who did that and succeeded, there are just as many who did that and failed.
Think about it, or better yet, Google it. You’ll be hard pressed to easily find a list of failures.
Even when you try to find people who have failed, you only find those who succeeded!
As an example, wearable technology is growing trend. The moment that clear winners emerge in the field, a rush of new entrepreneurs will come in to try and get their part of the pie. However, just because a product idea worked for someone else doesn’t mean it’ll work for you. Take an approach unique to your business.
SURVIVORSHIP BIAS CUSTOMER SERVICE
You’re a business, and you know that your success is tied to your relationship with your customers. That’s why when you get negative feedback, you’re eager to dig in and figure out what went wrong.
Studies have shown that the most vocal customers are the ones who’ll express their feelings. Everyone else will either give companies another chance or just leave.
Consider this, of the 90% of customers who don’t say anything, 78% of them leave. The 10% who complain are 90% more likely to stay. We’ve all heard the saying, “the squeaky wheel gets the grease” but let’s revise that thinking.
Instead of focusing on your unhappy customers, look at the behaviors of your happiest customers. Focus on the power users within your product because they’ll give you a model for success within your product. It’s also important to understand the difference between customer service and customer support. Having a solid grasp of how these two similar disciplines differ from one another is vital if you’re trying to gain insight into how your existing customers feel about your product.
It should be clear now that survivorship bias is everywhere we look. We usually get away with it—investing in building “the wrong” app sets you back financially but is a learning experience—but it’s still important to learn.
As David Cowan of Bessemer Venture Partners told Scientific American, “For every wealthy start-up founder, there are 100 other entrepreneurs who end up with only a cluttered garage.”
Like Abraham Wald, who saved hundreds of soldiers lives based on his simple observation, you take your blinders off. This could be all you need to do to save yourself frustration and prevent your business from failing.
Inspired by the book “How Not To Be Wrong” by Jordan Ellenberg
ABOUT ABRAHAM WALD
Wald was born in 1902 in the then Austria-Hungarian empire. After graduating in Mathematics he lectured in Economics in Vienna. As a Jew following the Anschluss between Nazi Germany and Austria in 1938 Wald and his family faced persecution and so they emigrated to the USA after he was offered a university position at Yale. During World War Two Wald was a member of the Statistical Research Group (SRG) as the US tried to approach military problems with research methodology.
Excerpts taken from mcdreeamiemusings.com