At Geckoboard, we’re on a mission to help people use data more simply and effectively. We’ve put together a series of quick lessons to help you spot fallacies in data or call out dodgy visualizations so you can use data with confidence and make better decisions both in work and in life.
All too often we see companies basing their strategies off what another company has done or off of one strategy that worked for them one time. Think about how many headlines you see that read something like “How StartupX grew 500% with this one email strategy.”
What you don’t see, however, is how many other companies tried that same exact email strategy and didn’t see any success. If anything, they lost money because they spent so much time on it with no results. Similarly, if you’re making a decision based on one strategy that worked one time for you but failed ten others, you’re making a decision based only on what survived and are ignoring data from failed attempts.
Turns out, we’re all comparing ourselves to each other and making important decisions using incomplete data. We’re only collecting information on those “worth writing about.”
This is called Survivorship Bias: the act of drawing conclusions from an incomplete set of data because that data has 'survived' some selection criteria.
Josh Pigford and the Baremetrics team did an excellent job illustrating this with “Most startups are not crushing it.” Josh wrote, “Ask any startup founder how things are going and they’ll tell you it’s ‘going great’. They’ll talk about some new feature they’re rolling out, or a new round of funding. Or maybe they’ll mention how much user growth they’ve had or that they just got covered in TechCrunch. All signs will point to ‘crushing it’.”
When Baremetrics analyzed their Live SaaS Benchmarks, they found that the average startup is making $19,000/month with 12 people on their team. And that every single year, most of those businesses are losing all of their customers. Yikes. That’s not exactly “crushing it.”
We need to be careful here too. Even Baremetrics’ data is only telling a part of a version of a story. They’ve only analyzed the metrics that survived their way into their platform.
The best thing you can do is make decisions for your business based on a complete picture of how your company is performing and how your customers are interacting with it. Let’s take a better look at what exactly Survivorship Bias is and how it could be impacting your business.
Survivorship Bias in history
When analyzing data, it’s important to ask yourself what data you don’t have. Sometimes the full picture is obscured because the data you do have has survived a selection of some sort. To illustrate, here’s a quick history lesson.
In World War II, a team was asked where the best place was to fit armor to a plane. The planes that came back from battle had bullet holes everywhere except the engine and cockpit. Counterintuitively, the team decided it was best to fit the armor where the data was showing there were fewer bullet holes.
They realized the odds of any part of the aircraft being hit were more or less constant. The reason they were seeing fewer aircraft with engine and cockpit damage was that those planes hadn’t survived. For more on the subject, I encourage you to read this excerpt from How Not To Be Wrong by Jordan Ellenberg.
When making a decision, make sure to consider whether the data you’re analyzing has survived anything and if there is any data potentially missing that should be considered.
How to avoid Survivorship Bias in your business
It’s your job to be aware of potential Survivorship Bias and how it could impact your business decisions. When conducting competitive analysis, know that you don’t have all of the data behind each competitor. And there were likely past competitors who haven’t survived! Try learning about those and what caused them to fail.
When conducting market analysis, gather all relevant data points on your customer market and include them all in your data analysis.
The same stands true for when you’re communicating with your team as a leader. To empower them to make the best decisions for your business, they need a complete picture of how your business is performing and the actions of your customers. Don’t only include the data that survived your selection process.
