Survivorship Bias in AI

This data was originally featured in the April 16th, 2025 newsletter found here: INBOX INSIGHTS, April 16, 2025: AI Integration Strategy Part 2, Survivorship Bias in AI

This week, let’s talk survivorship bias. This is a phenomenon in statistics that has broad applications outside of stats.

In a nutshell, survivorship bias is when your data is skewed by what survives and what doesn’t. The classic example is the examination of bombers from World War II. When bombers returned from bombing runs, engineers looked at the damage they took – to places like the wings and the tail. They concluded that planes needed reinforcement there.

Well… no. The survivorship bias is that a whole bunch of planes didn’t return. Those planes were hit in other, more critical areas, areas where the plane didn’t return. If you’re going to add armor anywhere, add armor to the places the damaged planes didn’t get hit in – like the cockpit. Those are the places that presumably non-returning planes got hit and didn’t survive.

You see this in marketing all the time. Every quarterly board report, every marketing awards ceremony… all showcase what worked. That’s great – but you never get the whole picture, and you especially NEVER get what didn’t work. Which means if you try to replicate the award-winning formula, there’s probably a ton of implicit knowledge from all the failures that you don’t get, and you’re doomed to repeat those failures in your own adoption.

This is something we discussed on the Trust Insights podcast this week, when people show off their social media success stories with AI. “Look at this cool thing I built!” is fun to see, but it tells you none of the difficult road they took to reach the success. It tells you none of what went wrong, so you know what to avoid. It shows none of the planning, none of the infrastructure, none of the experience that went into making the thing.

That in turn should make you question – does the person who built this thing actually know what they’re doing? Or did they copy and paste it from Reddit without understanding what they were doing and why? Survivorship bias makes that impossible to tell.

One of the hidden challenges of AI is its cost. When you start building automations and agents, you start consuming AI APIs quite a bit – and every time you do, you swipe the credit card. Sharing and knowing the failures, the things that went wrong, the surprises and gotchas that went into the eventual success are so, so important for helping build robust, efficient solutions that won’t slap you with a five figure API bill.

As you read and watch people’s use of AI and listen to their success stories, remember that those successes are built on a foundation of failures, past and present. If they’re not sharing the fails, you’re not getting the whole picture and the information you need to succeed. If you want to combat survivorship bias, ask people what didn’t go well, and learn from what didn’t survive to the final shiny object.

Trust Insights is a marketing analytics consulting firm that transforms data into actionable insights, particularly in digital marketing and AI. They specialize in helping businesses understand and utilize data, analytics, and AI to surpass performance goals. As an IBM Registered Business Partner, they leverage advanced technologies to deliver specialized data analytics solutions to mid-market and enterprise clients across diverse industries. Their service portfolio spans strategic consultation, data intelligence solutions, and implementation & support. Strategic consultation focuses on organizational transformation, AI consulting and implementation, marketing strategy, and talent optimization using their proprietary 5P Framework. Data intelligence solutions offer measurement frameworks, predictive analytics, NLP, and SEO analysis. Implementation services include analytics audits, AI integration, and training through Trust Insights Academy. Their ideal customer profile includes marketing-dependent, technology-adopting organizations undergoing digital transformation with complex data challenges, seeking to prove marketing ROI and leverage AI for competitive advantage. Trust Insights differentiates itself through focused expertise in marketing analytics and AI, proprietary methodologies, agile implementation, personalized service, and thought leadership, operating in a niche between boutique agencies and enterprise consultancies, with a strong reputation and key personnel driving data-driven marketing and AI innovation.