The increasing prevalence of A/B testing has revealed a powerful but seldom discussed insight: Our intuitions are soberingly bad. Yet most of us are either unaware or choose not to believe this. Time and time again, randomized controlled trials reveal that our predictions on the outcomes of business decisions are deeply fallible. Experiments show that our optimism in new features is often misguided, what we thought would help actually hurts, and what we believed to be the next big thing fails completely. Our intuitions can even lead to worse outcomes than what we would expect from random chance alone.
This is not a talk about A/B testing but rather about what empiricism reveals about our intuitions. Eric Colson and Daragh Sibley describe some types of errors domain experts commit, explain how these systematic errors arise from heuristic reasoning processes that served us well on an evolutionary time scale but now impair our judgments, and illustrate how cognitive heuristics can compromise decisions and their evaluation.
While we can’t change how our brains are wired, knowledge of how they work can give us an advantage. Eric and Daragh share several mechanisms to mitigate these human limitations and improve our decision making and argue that businesses that make fewer bad decisions will have a distinct competitive advantage.
Eric Colson is chief algorithms officer at Stitch Fix, where he leads a team of 100+ data scientists and is responsible for the multitude of algorithms that are pervasive to nearly every function of the company, from merchandise, inventory, and marketing to forecasting and demand, operations, and the styling recommender system. He’s also an advisor to several big data startups. Previously, Eric was vice president of data science and engineering at Netflix. He holds a BA in economics from SFSU, an MS in information systems from GGU, and an MS in management science and engineering from Stanford.
Daragh Sibley is director of data science at Stitch Fix, where he leads a team of data scientists that use algorithms and the scientific method to optimize the portfolio of products stocked in Stitch Fix’s inventory. Previously, Daragh spent a decade in academia, where he developed neural networks of human language acquisition and tested their predictions with behavioral and neuroimaging experiments.
Comments on this page are now closed.
©2019, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • firstname.lastname@example.org