Audi's journey to an enterprise big data platform

Description

Carsten Herbe and Matthias Graunitz detail Audi’s journey from a Hadoop proof of concept to a multitenant enterprise platform, sharing lessons learned, the decisions Audi made, and how a number of use cases are implemented using the platform.

During the process of setting up the big data infrastructure, Audi often had to find the right balance between enterprise integration and speed (e.g., whether to use the existing Active Directory for both LDAP and KDC or set up its own KDC). Using a shared enterprise service like Active Directory requires following certain naming conventions and restricted access whereas running your own KDC brings much more flexibility but also adds another component to maintain. Carsten and Matthias explore the advantages and disadvantages and explain why Audi chose its approach. For data ingestion of both batch and streaming data, Audi’s platform uses Apache Kafka. Carsten and Matthias explain their team installed a separated Kafka cluster from Audi’s Hortonworks platform and discuss the pros and cons of using the Kafka binary protocol and the HTTP REST protocol—not only from a technical perspective but also from the organizational perspective.

Although Audi has already achieved quite a lot, its journey has not yet ended. There are still some open topics to address, such as providing a unified logging solution for applications spanning multiple platforms, finally offering a notebook like Zeppelin to its analysts, which will require an upgrade to the next HDP release, and addressing legal issues like GDPR. Carsten and Matthias conclude with a short glimpse into the ongoing extension of the on-premises platform into a hybrid cloud platform.