Running multidisciplinary big data workloads in the cloud
Who is this presentation for?
- Data engineers, data scientists, BI engineers, analytic engineers, and those in IT
Organizations now run diverse, multidisciplinary, big data workloads that span data engineering, data warehousing, and data science applications. Many of these workloads operate on the same underlying data, and the workloads themselves can be transient or long running in nature.
There are many challenges with moving these workloads to the cloud. Jason Wang, Tony Wu, and Vinithra Varadharajan take a technical deep dive into cloud architecture and the challenges in moving to the cloud, including things to keep in mind when moving the cloud and why it may not be as simple as you thought (e.g., data migration and duplication between on-premises and in the cloud), core cloud paradigms not present on-premises that drive architecture decisions (e.g., bursting and different cluster lifecycles and tenancy), security best practices in the cloud (e.g., the basics, common pitfalls, and things often overlooked that you need to get right), and how to manage metadata between various workloads across multiple clusters, both on-premises and in the cloud.
You’ll get your hands dirty and learn how to successfully set up and run a data pipeline in the cloud that integrates with data engineering and data warehousing workflows using the Cloudera Altus PaaS offering, powered by Cloudera Altus SDX, to run various big data workloads. Jason, Tony, and Vinithra explore considerations and best practices in getting data pipelines running. Along the way, you’ll see how to share metadata across workloads in a big data architecture.
- Familiarity with public cloud concepts
- A basic understanding of big data workloads (data engineering and data warehousing)
Materials or downloads needed in advance
- A WiFi-enabled laptop (If you want to use the CLI, you need to have Python 3.6 installed and have terminal access.)
What you'll learn
- Learn how to successfully run a data analytics pipeline in the cloud and integrate data engineering and data analytic workflows
- Understand considerations and best practices for data analytics pipelines in the cloud
- Explore approaches for sharing metadata across workloads in a big data PaaS
Jason Wang is a software engineer at Cloudera focusing on the cloud.
Tony Wu is an engineering manager at Cloudera, where he manages the Altus core engineering team. Previously, Tony was a team lead for the partner engineering team at Cloudera. He’s responsible for Microsoft Azure integration for Cloudera Director.
Vinithra Varadharajan is a senior engineering manager in the cloud organization at Cloudera, where she’s responsible for the cloud portfolio products, including Altus Data Engineering, Altus Analytic Database, Altus SDX, and Cloudera Director. Previously, Vinithra was a software engineer at Cloudera working on Cloudera Director and Cloudera Manager with a focus on automating Hadoop lifecycle management.
Leave a Comment or Question
Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?
Join the conversation here (requires login)
For conference registration information and customer service
For more information on community discounts and trade opportunities with O’Reilly conferences
For information on exhibiting or sponsoring a conference
View a complete list of Strata Data Conference contacts