Over the past several years, ever-increasing quantities of data are being processed within public clouds. The cloud promises to provide solutions to some of the limitations of conventional single multipurpose clusters offering hyperscale storage decoupled from elastic, on-demand compute and allows data to be shared between on-demand provisioned processing engines such as Hive, Spark, and Impala. But to fulfill this promise, you first need to solve several technical challenges: simple resource allocation, cross-cluster metadata sharing, and a common authorization framework. Without comprehensive answers to these questions, the challenges of single cluster model are simply duplicated inside a public cloud environment.
The cloud enables the delivery of solutions to single, multipurpose clusters offering hyperscale storage decoupled from elastic, on-demand computing. Vinithra Varadharajan, Jason Wang, Eugene Fratkin, and Mael Ropars detail new paradigms to effectively run production-level pipelines with minimal operational overhead. As a part of the deep dive, they also walk you through creating such a pipeline and executing data processing and data analytic workflows. Join in to learn how to remove barriers to data discovery, metadata sharing, and access control.
Eugene Fratkin is a director of engineering at Cloudera, heading Cloud R&D. He was one of the founding members of the Apache MADlib project (scalable in-database algorithms for machine learning). Previously, Eugene was a cofounder of a Sequoia Capital-backed company focusing on applications of data analytics to problems of genomics. He holds PhD in computer science from Stanford University’s AI lab.
Vinithra Varadharajan is a senior engineering manager in the cloud organization at Cloudera, where she’s responsible for the cloud portfolio products, including Altus Data Engineering, Altus Analytic Database, Altus SDX, and Cloudera Director. Previously, Vinithra was a software engineer at Cloudera working on Cloudera Director and Cloudera Manager with a focus on automating Hadoop lifecycle management.
Mael Ropars is a senior sales engineer at Cloudera, helping customers solve their big data problems using enterprise data hubs based on Hadoop. Mael has 15 years’ experience working around big data, information management, and middleware in technical sales and service delivery.
Jason Wang is a software engineer at Cloudera focusing on the cloud.
©2018, O’Reilly UK Ltd • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • firstname.lastname@example.org