Carsten Herbe and Matthias Graunitz detail Audi’s journey from a Hadoop proof of concept to a multitenant enterprise platform, sharing lessons learned, the decisions Audi made, and how a number of use cases are implemented using the platform.
During the process of setting up the big data infrastructure, Audi often had to find the right balance between enterprise integration and speed (e.g., whether to use the existing Active Directory for both LDAP and KDC or set up its own KDC). Using a shared enterprise service like Active Directory requires following certain naming conventions and restricted access whereas running your own KDC brings much more flexibility but also adds another component to maintain. Carsten and Matthias explore the advantages and disadvantages and explain why Audi chose its approach. For data ingestion of both batch and streaming data, Audi’s platform uses Apache Kafka. Carsten and Matthias explain their team installed a separated Kafka cluster from Audi’s Hortonworks platform and discuss the pros and cons of using the Kafka binary protocol and the HTTP REST protocol—not only from a technical perspective but also from the organizational perspective.
Although Audi has already achieved quite a lot, its journey has not yet ended. There are still some open topics to address, such as providing a unified logging solution for applications spanning multiple platforms, finally offering a notebook like Zeppelin to its analysts, which will require an upgrade to the next HDP release, and addressing legal issues like GDPR. Carsten and Matthias conclude with a short glimpse into the ongoing extension of the on-premises platform into a hybrid cloud platform.
Carsten Herbe is a big data architect at Audi Business Innovation GmbH, a subsidiary of Audi focused on developing new mobility services and innovative IT solutions, where he is helping build a big data platform based on Hadoop and Kafka and as an solution architect, is responsible for developing and running the first analytical applications on that platform. Carsten has more than 10 years’ experience delivering data warehouse and BI solutions as well as big data infrastructure and solutions.
Matthias Graunitz is a big data architect at Audi’s Competence Center for Big Data and Business Intelligence, where he is responsible for the architectural framework of the Hadoop ecosystem, a separate Kafka cluster, and the data science tool kits provided by the Center of Competence for all business departments at Audi. Matthias has more than 10 years’ experience in the field of business intelligence and big data.
Comments on this page are now closed.
©2018, O’Reilly UK Ltd • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • email@example.com