Running multidisciplinary big data workloads in the cloud with CDP

Description

Organizations now run diverse, multidisciplinary, big data workloads that span data engineering, data warehousing, and data science applications. Many of these workloads operate on the same underlying data, and the workloads themselves can be transient or long running in nature.
There are many challenges with moving these workloads to the cloud. In this talk we start off with a technical deep dive into Cloudera Data Platform (CDP).

Topics include:

• Architecture of CDP and considerations that went into key design decisions (e.g. bursting, security, and different workload lifecycles)
• Options for data engineering, machine learning, data warehouse as well as the different capabilities, including autoscaling, for each option
• Security, governance, and metadata best practices in the cloud (e.g. the basics, common pitfalls, and things often overlooked that you need to get right)
• Delivering data science capabilities to a wide range of users, without allocating individual clusters.

In addition, you’ll see how to successfully set up and run a data pipeline with CDP that integrates both data engineering and data warehousing workflows. We’ll explore considerations and best practices in getting data pipelines running. Along the way you’ll also see how to share metadata across workloads in a big data architecture. Note: This session will not have a hands-on component, as CDP is in Preview.