Skip to main content

Apache Spark: A Killer or Savior of Apache Hadoop?

Roman Shaposhnik (Pivotal Inc.)
Slides:   1-PDF 

The Big Boss™ has just OKed the first Hadoop cluster in the company. You are the guy in charge of analyzing petabytes of your company’s valuable data using a combination of custom MapReduce jobs and SQL-on-Hadoop solutions (you are using HAWQ aren’t you?). All of a sudden the web is full of articles telling you that Hadoop is dead, Spark has won and you should quit while you’re still ahead. But should you?

The truth is, Apache Spark has quickly become one of the most active projects in all of Apache Software Foundation attracting more contributors in the past 6 months that Hadoop MapReduce. Of course, it is also true, that those new and exciting tools for data processing and machine learning from Spark operate within the same Hadoop ecosystem you’ve fallen in love with.

In this talk we’ll introduce the core ideas behind Spark engine, look into how it builds off of Hadoop’s scheduling capabilities and how the generality of core Spark helps unify operational backend for current and future data management tools built for Hadoop. You will also be invited to a whirlwind tour of higher-level libraries built on Spark like Shark for SQL, MLib for machine learning, and Spark Streaming.

This is your chance to understand where Hadoop ends, Spark begins and how the two will co-exist making your existing investment in Hadoop cluster pay off even more.

This session is sponsored by Pivotal

Photo of Roman Shaposhnik

Roman Shaposhnik

Pivotal Inc.

Roman Shaposhnik is a committer on Apache Hadoop, and holds a Chair position for the Apache Bigtop and Apache Incubator projects. Roman has been involved in Open Source software for more than a decade and has hacked projects ranging from the Linux kernel to the flagship multimedia library known as FFmpeg. He grew up in Sun Microsystems where he had an opportunity to learn from the best software engineers in the industry. Roman’s alma mater is St. Petersburg State University, Russia where he studied to be a mathematician.