Presented By
O’Reilly + Cloudera
Make Data Work
March 25-28, 2019
San Francisco, CA
Please log in

Cloud native data pipelines with Apache Kafka

Gwen Shapira (Confluent)
5:10pm–5:50pm Wednesday, March 27, 2019
Data Engineering & Architecture
 Location: 2001
Secondary topics:  AI and Data technologies in the cloud, Data Integration and Data Pipelines
Average rating: ****.
(4.64, 11 ratings)

Who is this presentation for?

  • Data engineers and architects

Level

Intermediate

Prerequisite knowledge

  • Familiarity with Kafka and public clouds (useful but not required)

What you'll learn

  • Learn how to design flexible data pipelines, interact with APIs and serverless functions, and integrate stateful and stateless applications

Description

As microservices, data services, and serverless APIs proliferate in a cloud native world, analysts still need to report on the business as a whole. Data engineers need to collect and standardize data in an increasingly complex and diverse system. Luckily, the problem is also the solution. The way to manage data in a cloud native environment is to build cloud native data pipelines.

Gwen Shapira discusses how data engineering requirements have changed in a cloud native world and how the solutions have changed with them. She then shares architectural patterns that are commonly used to build cloud native data infrastructure and explains how they help you build flexible, scalable, and reliable pipelines to give your business visibility on all your data.

Photo of Gwen Shapira

Gwen Shapira

Confluent

Gwen Shapira is a system architect at Confluent, where she helps customers achieve success with their Apache Kafka implementations. She has 15 years of experience working with code and customers to build scalable data architectures, integrating relational and big data technologies. Gwen currently specializes in building real-time reliable data processing pipelines using Apache Kafka. Gwen is an Oracle Ace Director, the coauthor of Hadoop Application Architectures, and a frequent presenter at industry conferences. She is also a committer on Apache Kafka and Apache Sqoop. When Gwen isn’t coding or building data pipelines, you can find her pedaling her bike, exploring the roads and trails of California and beyond.