Presented By O’Reilly and Cloudera
Make Data Work
21–22 May 2018: Training
22–24 May 2018: Tutorials & Conference
London, UK

Elastic map matching using Cloudera Altus and Apache Spark

Timo Graen (Volkswagen AG ), Robert Neumann (Ultra Tendency)
14:5515:35 Wednesday, 23 May 2018
Data engineering and architecture
Location: S11B Level: Beginner
Secondary topics:  Transportation and Logistics

Who is this presentation for?

  • Data engineers and those working with geodata

Prerequisite knowledge

  • Familiarity with the Cloudera ecosystem and Spark

What you'll learn

  • Learn how Volkswagen Commercial Vehicle uses map matching for geodata and Altus to scale load over hybrid clouds
  • Understand how to create an elastic map-matching system

Description

Map matching describes the problem of mapping GPS coordinates that were recorded through a device to a geographical reference system, such as a map. Map-matching applications exist in almost every telematics use case and are therefore crucial to car manufacturers, such as Volkswagen.

Volkswagen Commercial Vehicle’s Telematics Division offers analytical services to commercial customers, based on map-matched position data. These services include route optimization, smart service propositioning, and a driver’s logbook. Throughout a day the position data of a vehicle (e.g., the new Volkswagen Crafter) is collected into a central data store. A nightly running Spark batch job picks up the data from the central data store and applies the map-matching operation to every vehicle’s position. The map-matched positions are then written back to the central data store and from there fed into downstream processing systems.

The Spark-based map-matching application is submitted as an Altus job to an on-demand Cloudera cluster. The on-demand cluster gets automatically provisioned through Altus prior to submitting the Spark job and is released right after the map matching job has finished. This way Altus enables elastics resource provisioning and allows for acquiring compute resources only for as long as they are really needed, resulting in major savings. As an additional benefit, Altus allows for the compute cluster’s size growing in accordance to the amount of data that needs to be map-matched.

Timo Graen and Robert Neumann detail the architecture behind Volkswagen Commercial Vehicle’s Altus-based map-matching application and lead a live demo featuring a map matching job in Altus.

Timo Graen

Volkswagen AG

Photo of Robert Neumann

Robert Neumann

Ultra Tendency

Robert Neumann is founder and CEO of Ultra Tendency. He has more than a decade of experience designing and developing Hadoop-based applications and has been using Spark since v0.9.