Running multidisciplinary big data workloads in the cloud with CDP
Who is this presentation for?
- Data engineers, data scientists, BI engineers, analytic engineers, and those in IT
Organizations now run diverse, multidisciplinary, big data workloads that span data engineering, data warehousing, and data science applications. Many of these workloads operate on the same underlying data, and the workloads themselves can be transient or long running in nature.
There are many challenges with moving these workloads to the cloud. In this talk we start off with a technical deep dive into Cloudera Data Platform (CDP).
- Architecture of CDP and considerations that went into key design decisions (e.g. bursting, security, and different workload lifecycles)
- Options for data engineering, machine learning, data warehouse as well as the different capabilities, including autoscaling, for each option
- Security, governance, and metadata best practices in the cloud (e.g. the basics, common pitfalls, and things often overlooked that you need to get right)
- Delivering data science capabilities to a wide range of users, without allocating individual clusters.
In addition, you’ll see how to successfully set up and run a data pipeline with CDP that integrates both data engineering and data warehousing workflows. We’ll explore considerations and best practices in getting data pipelines running. Along the way you’ll also see how to share metadata across workloads in a big data architecture. Note: This session will not have a hands-on component, as CDP is in Preview.
- Familiarity with public cloud concepts
- A basic understanding of big data workloads (data engineering and data warehousing)
Materials or downloads needed in advance
- A WiFi-enabled laptop (If you want to use the CLI, you need to have Python 3.6 installed and have terminal access.)
What you'll learn
- Learn how to successfully run a data analytics pipeline in the cloud and integrate data engineering and data analytic workflows
- Understand considerations and best practices for data analytics pipelines in the cloud
- Explore approaches for sharing metadata across workloads in a big data PaaS
James Morantus is a Cloud Solutions/Customer Success Engineer at Cloudera. Previously, James was a Senior Solutions Architect with the Professional Service organization at Cloudera, delivering services both on-prem and on the public cloud.
Naren Koneru is an engineering manager at Cloudera and leads the navigator development team. Prior to Cloudera, Naren was at Miti, building enterprise-wide metadata and governance solutions. Before joining Miti, Naren spent over seven years with the platform team at Informatica and was instrumental in making PowerCenter the leading data integration platform. He has a master’s in computer science from East Tennessee State University and bachelor’s from Osmania University.
Leave a Comment or Question
Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?
Join the conversation here (requires login)
For conference registration information and customer service
For more information on community discounts and trade opportunities with O’Reilly conferences
For information on exhibiting or sponsoring a conference
For media/analyst press inquires