Public cloud usage for large-scale data processing is rapidly increasing, and running data engineering workloads in the cloud is becoming easier and more cost effective. Compute engines have adapted to leverage cloud infrastructure, including object storage and elastic compute. For example, Hive, Spark, Impala, and HBase compute engines are able to read input from and write output directly to AWS S3 and Azure Data Lake storage. Moreover, these read and write paths have been optimized for fast processing speeds, lowering the overall cost of running a job. In addition, platform-as-a-service offerings for data processing in the cloud have evolved to minimize the operational overhead of clusters, enabling end users to focus on developing, running, and troubleshooting jobs.
It is important for end users to be able to implement data pipeline workflows that seamlessly transition from one stage of the data pipeline to the next. Aishwarya Venkataraman, Jason Wang, Mala Ramakrishnan, Stefan Salandy, and Vinithra Varadharajan lead a deep dive into running data analytic workloads in a managed service capacity in the public cloud and highlight cloud infrastructure best practices.
Jason is a software engineer at Cloudera focusing on the cloud.
Mala Ramakrishnan heads product initiatives for Cloudera Altus – big data platform-as-a-service. She has 17+ years experience in product management, marketing, and software development in organizations of varied sizes that deliver middleware, software security, network optimization, and mobile computing. She holds a master’s degree in computer science from Stanford University.
Cloudera Systems Engineer
Vinithra Varadharajan is an engineering manager in the cloud organization at Cloudera, where she is responsible for products such as Cloudera Director and Cloudera’s usage-based billing service. Previously, Vinithra was a software engineer at Cloudera, working on Cloudera Director and Cloudera Manager with a focus on automating Hadoop lifecycle management.
Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?
Join the conversation here (requires login)
©2018, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • firstname.lastname@example.org