In early 2016, Spotify decided that it didn’t want to be in the data center business. The future was the cloud. Josh Baer and Alison Gilles explain what it took to move Spotify to the cloud, covering Spotify’s technology choices, challenges faced, and the lessons Spotify learned along the way.
Before migrating to Google Cloud Platform (GCP), data processing at Spotify was mainly done in Hadoop with products in the larger Apache Hadoop ecosystem: Hive for ad hoc analysis, MapReduce for daily batch jobs, Storm for real-time processing, and Spark for machine learning. Josh and Alison describe how Spotify’s data processing platform has evolved through the cloud migration to incorporate GCP offerings like BigQuery, Cloud Dataflow, Cloud Pub/Sub, and TensorFlow.
Josh and Alison also explore some of the organizational changes and culture shifts that the cloud migration has brought—training highly skilled engineers, who are used to solving their own problems, in how to problem solve alongside a provider; leveraging Spotify’s relationship with Google as a vendor; and leading an engineering organization through a transition to focus higher up the stack—as well as some of the beneficial and not-so-beneficial changes the company has made during the still-in-progress migration.
Josh Baer is the machine learning platform lead at Spotify, building out the tools, processing, and infrastructure for robust ML experiences; enabling teams to leverage ML and AI sustainably in their products, research, and services; and providing a cohesive experience. Previously, Josh led the Hadoop and stream processing teams.
Alison Gilles is director of engineering for data infrastructure at Spotify, where she coaches and leads teams in backend services and data infrastructure. Previously, she led engineering teams at nonprofit organizations in education and corporate social responsibility.
Comments on this page are now closed.
©2017, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • firstname.lastname@example.org