Presented By O’Reilly and Cloudera
Make Data Work
21–22 May 2018: Training
22–24 May 2018: Tutorials & Conference
London, UK

Audi's journey to an enterprise big data platform

Carsten Herbe (Audi Business Innovation GmbH), Matthias Graunitz (Audi AG)
14:0514:45 Wednesday, 23 May 2018
Data engineering and architecture
Location: S11B Level: Intermediate
Secondary topics:  Data Platforms, Transportation and Logistics

Who is this presentation for?

  • Anyone building or using an enterprise big data platform

Prerequisite knowledge

  • A basic understanding of Hadoop, Kafka, data warehousing, and business intelligence

What you'll learn

  • Explore how Audi built an enterprise big data platform

Description

Carsten Herbe and Matthias Graunitz detail Audi’s journey from a Hadoop proof of concept to a multitenant enterprise platform, sharing lessons learned, the decisions Audi made, and how a number of use cases are implemented using the platform.

During the process of setting up the big data infrastructure, Audi often had to find the right balance between enterprise integration and speed (e.g., whether to use the existing Active Directory for both LDAP and KDC or set up its own KDC). Using a shared enterprise service like Active Directory requires following certain naming conventions and restricted access whereas running your own KDC brings much more flexibility but also adds another component to maintain. Carsten and Matthias explore the advantages and disadvantages and explain why Audi chose its approach. For data ingestion of both batch and streaming data, Audi’s platform uses Apache Kafka. Carsten and Matthias explain their team installed a separated Kafka cluster from Audi’s Hortonworks platform and discuss the pros and cons of using the Kafka binary protocol and the HTTP REST protocol—not only from a technical perspective but also from the organizational perspective.

Although Audi has already achieved quite a lot, its journey has not yet ended. There are still some open topics to address, such as providing a unified logging solution for applications spanning multiple platforms, finally offering a notebook like Zeppelin to its analysts, which will require an upgrade to the next HDP release, and addressing legal issues like GDPR. Carsten and Matthias conclude with a short glimpse into the ongoing extension of the on-premises platform into a hybrid cloud platform.

Photo of Carsten Herbe

Carsten Herbe

Audi Business Innovation GmbH

Carsten Herbe is a big data architect at Audi Business Innovation GmbH, a subsidiary of Audi focused on developing new mobility services and innovative IT solutions, where he is helping build a big data platform based on Hadoop and Kafka and as an solution architect, is responsible for developing and running the first analytical applications on that platform. Carsten has more than 10 years’ experience delivering data warehouse and BI solutions as well as big data infrastructure and solutions.

Photo of Matthias Graunitz

Matthias Graunitz

Audi AG

Matthias Graunitz is a big data architect at Audi, where he works at the company’s Competence Center for Big Data and Business Intelligence, where he is responsible for the architectural framework of the Hadoop ecosystem, a separate Kafka Cluster as well as for the data science tool kits provided by the Center of Competence for all business departments at Audi. Matthias has more than 10 years’ experience in the field of business intelligence and big data.

Leave a Comment or Question

Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?

Join the conversation here (requires login)