Scaling Apache Spark at Facebook

Description

Spark started at Facebook as an experiment when the project was still in its early phases. Spark’s appeal stemmed from its ease of use and an integrated environment to run SQL, MLlib, and custom applications. At that time, the system was used by a handful of people to process small amounts of data.

However, Facebook has come a long way since then. Currently, Spark is one of Facebook’s primary SQL engines in addition to being the primary system for writing custom batch applications. Sameer Agarwal dives into the story of how Facebook optimized, tuned, and scaled Apache Spark to run on clusters of tens of thousands of machines, processing hundreds of petabytes of data, and being used by thousands of data scientists, engineers, and product analysts every day. You’ll specifically hear about scaling compute, or how Facebook runs Spark efficiently and reliably on tens of thousands of heterogenous machines in disaggregated (shared storage) clusters; optimizing core engine, or how Facebook continuously tunes, optimizes, and adds features to the core engine in order to maximize the useful work done per second; and scaling users, or how Facebook makes Spark easy to use and faster to debug to seamlessly onboard new users.