Presented By O’Reilly and Cloudera
Make Data Work
21–22 May 2018: Training
22–24 May 2018: Tutorials & Conference
London, UK

Audi's journey to an enterprise big data platform

Carsten Herbe (Audi Business Innovation GmbH), Matthias Graunitz (Audi AG)
14:0514:45 Wednesday, 23 May 2018
Data engineering and architecture
Location: S11B Level: Intermediate
Secondary topics:  Data Platforms, Transportation and Logistics
Average rating: *****
(5.00, 1 rating)

Who is this presentation for?

  • Anyone building or using an enterprise big data platform

Prerequisite knowledge

  • A basic understanding of Hadoop, Kafka, data warehousing, and business intelligence

What you'll learn

  • Explore how Audi built an enterprise big data platform

Description

Carsten Herbe and Matthias Graunitz detail Audi’s journey from a Hadoop proof of concept to a multitenant enterprise platform, sharing lessons learned, the decisions Audi made, and how a number of use cases are implemented using the platform.

During the process of setting up the big data infrastructure, Audi often had to find the right balance between enterprise integration and speed (e.g., whether to use the existing Active Directory for both LDAP and KDC or set up its own KDC). Using a shared enterprise service like Active Directory requires following certain naming conventions and restricted access whereas running your own KDC brings much more flexibility but also adds another component to maintain. Carsten and Matthias explore the advantages and disadvantages and explain why Audi chose its approach. For data ingestion of both batch and streaming data, Audi’s platform uses Apache Kafka. Carsten and Matthias explain their team installed a separated Kafka cluster from Audi’s Hortonworks platform and discuss the pros and cons of using the Kafka binary protocol and the HTTP REST protocol—not only from a technical perspective but also from the organizational perspective.

Although Audi has already achieved quite a lot, its journey has not yet ended. There are still some open topics to address, such as providing a unified logging solution for applications spanning multiple platforms, finally offering a notebook like Zeppelin to its analysts, which will require an upgrade to the next HDP release, and addressing legal issues like GDPR. Carsten and Matthias conclude with a short glimpse into the ongoing extension of the on-premises platform into a hybrid cloud platform.

Photo of Carsten Herbe

Carsten Herbe

Audi Business Innovation GmbH

Carsten Herbe is a big data architect at Audi Business Innovation GmbH, a subsidiary of Audi focused on developing new mobility services and innovative IT solutions, where he is helping build a big data platform based on Hadoop and Kafka and as an solution architect, is responsible for developing and running the first analytical applications on that platform. Carsten has more than 10 years’ experience delivering data warehouse and BI solutions as well as big data infrastructure and solutions.

Photo of Matthias Graunitz

Matthias Graunitz

Audi AG

Matthias Graunitz is a big data architect at Audi’s Competence Center for Big Data and Business Intelligence, where he is responsible for the architectural framework of the Hadoop ecosystem, a separate Kafka cluster, and the data science tool kits provided by the Center of Competence for all business departments at Audi. Matthias has more than 10 years’ experience in the field of business intelligence and big data.

Comments on this page are now closed.

Comments

Picture of Carsten Herbe
Carsten Herbe | BIG DATA ARCHITECT
3/07/2018 10:11 BST

Publishing the slides is handled by the conference team, but I expect this to happen soon …

Alexander Rodriguez | BIG DATA PROJECT MANAGER
2/07/2018 13:25 BST

Hello, thanks for yout interesting presentation ¿it is already published?

Picture of Matthias Graunitz
Matthias Graunitz | BIG DATA ARCHITECT
24/05/2018 13:04 BST

The slides will be published at the Conference Website

Mohamed Maamoun | SENIOR ANALYTICS PLATFORM ARCHITECT
24/05/2018 9:47 BST

Hi, thank you for the interesting presentation, I found it very helpful.
Is there a way to get the Slides in good quality? Thanks