Presented By O'Reilly and Cloudera
Make Data Work
Feb 17–20, 2015 • San Jose, CA

The Evolution of Hadoop at Spotify- Through Failures and Pain

Josh Baer (Spotify), Rafał Wojdyła (Spotify)
10:40am–11:20am Thursday, 02/19/2015
Hadoop in Action
Location: 210 A/E
Average rating: ****.
(4.25, 4 ratings)
Slides:   1-PDF 

The quickest way to learn and evolve infrastructure is by encountering obstacles and being forced to overcome limitations that keep you inches away from project goals. At Spotify, we’ve encountered many of these obstacles and frustrations as we grew our Hadoop cluster from a few machines in an office closet aggregating played song events for financial reports, to our current 900 node cluster that plays a large role in many features that you see in our application today.

Two members of Spotify’s Hadoop ‘squad’ will weave in war stories, failures, frustrations and lessons learned to describe the Hadoop/Big Data architecture at Spotify and talk about how that architecture has evolved.

We’ll talk about how and why we use a number of tools, including Apache Falcon and Apache Bigtop to test changes; Apache Crunch, Scalding and Hive w/ Tez to build features and provide analytics; and Snakebite and Luigi, two in-house tools created to overcome common frustrations.

Photo of Josh Baer

Josh Baer


Josh spent six years as a software engineer building infrastructure components at AT&T before discovering the world of ‘Big Data’ in a class at NYU by O’Reilly author Foster Provost. He ‘joined the band’ at Spotify in early 2013 and has worked on a small team focusing on stabilizing and enhancing the Hadoop infrastructure, performing multiple migrations, upgrades and growing the cluster from 190 nodes to over 900. Today, Josh lives in Stockholm, Sweden and leads the platform vision as the Hadoop Product Owner.

Josh holds a BS in Computer Science/Philosophy from the University of Pittsburgh and a MS in Computer Science from NYU.

Photo of Rafał Wojdyła

Rafał Wojdyła


Rafal is an engineer at Spotify, a member of Hadoop squad responsible for operating, maintaing and growing one of the biggest Hadoop cluster in Europe. He is a core committer to snakebite – super fast pure python HDFS client.