4–7 Nov 2019

Evolution of a modern cloud-based data lake

15:0015:45 Thursday, 7 November 2019
Location: M8
Secondary topics:  Best Practice, Case Study

Who is this presentation for?

  • Data engineers, data architects, and product specialists in data

Level

Intermediate

Description

Viacheslav Inozemtsev outlines the experience of building and evolving the cloud-based data lake of a company as large as Zalando. In particular, he addresses the three main areas of ingestion of data from all the various sources in the company, easy and convenient access to data, and security and governance at the scale of more than 100 teams. He also explores the issue of cost throughout all three parts.

The first challenge—ingestion of data—is a broad topic on its own. Viacheslav examines the evolution of Zalando’s ingestion pipelines from different company-wide data sources, such as messaging bus, data warehouse, Google Analytics platform, as well as custom datasets on demand. For the second challenge—access to data—you’ll learn the evolution of the tools and principles Zalando developed to give the rest of the company convenient means to consume data and extract information from it. The largest challenge is security and governance, although it doesn’t bring any value directly. Viacheslav explores how the company addressed security and access management in the first place and how it evolved them later when better frameworks and services appeared on the market.

Prerequisite knowledge

  • A basic understanding of data engineering and cloud technologies

What you'll learn

  • Learn best practices in building and evolving a data lake, the benefits of using cloud for building a data lake, and key factors of success while building a data lake
Photo of Viacheslav Inozemtsev

Viacheslav Inozemtsev

Zalando

Viacheslav Inozemtsev is a data engineer at Zalando on its data lake project, building an internal data platform on top of S3, Spark, Presto, and serverless cloud technologies, enabling machine learning and AI for all teams and departments of the company, and solving GDPR in one place. He has 7+ years of data and software engineering experience. He earned a degree in applied mathematics and an MSc degree in computer science mostly on the topics of data processing and analysis.

Leave a Comment or Question

Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?

Join the conversation here (requires login)

Contact us

confreg@oreilly.com

For conference registration information and customer service

partners@oreilly.com

For more information on community discounts and trade opportunities with O’Reilly conferences

Become a sponsor

For information on exhibiting or sponsoring a conference

pr@oreilly.com

For media/analyst press inquires