Presented By O'Reilly and Cloudera
Make Data Work
December 1–3, 2015 • Singapore

Application of Spark on analyzing massive GIS data for a large number of mobile objects

Masaru Dobashi (NTT DATA Corporation), Yoshitaka Suzuki (IHI Corporation)
2:20pm–3:00pm Wednesday, 12/02/2015
IoT and Real-time
Location: 324 Level: Intermediate
Tags: iot, geo
Average rating: ***..
(3.43, 7 ratings)

Prerequisite Knowledge

Basic knowledge about Hadoop and related products.

Description

We are developing a platform to process massive sensor data obtained from social infrastructures and industrial machinery all over the world, in order to achieve advanced safety management. To provide service for mobile objects, we are implementing into this platform algorithms and software to analyze large-scale GIS data.

We first focused on the analysis of time-series using over 100,000 mobile objects. Since the number of observations exceeds 1.5 billion records per year, the algorithm requires both scalability and flexibility. To address this issue, we evaluated Spark on YARN as a scalable and flexible platform.

In this session, we’ll talk about the capability of Spark to realize numerical data processing, the best practices for application development, and realistic lessons on operating Spark on YARN.

IHI is a leading manufacturer in Japan that provides a wide variety of products including energy plants and aircraft engines. IHI is addressing this project, and NTT Data is providing technical support for Spark on YARN infrastructure building.

Photo of Masaru Dobashi

Masaru Dobashi

NTT DATA Corporation

Masaru Dobashi is a system infrastructure engineer and leads the OSS professional service team at NTT DATA Corporation. Masaru developed an enterprise Hadoop cluster consisting of over 1,000 nodes in 2009, which was one of the largest Hadoop clusters in Japan at the time. After that, he designed and provisioned several kinds of clusters using nonHadoop OSS, such as Spark and Storm. Masaru is now responsible for introducing Hadoop, Spark, Storm, and other OSS middlewares into enterprise systems and developing data processing systems.

Photo of Yoshitaka Suzuki

Yoshitaka Suzuki

IHI Corporation

Yoshitaka Suzuki is a researcher in information science and technology at IHI Corporation. Yoshitaka has developed anomaly detection algorithms for several kinds of products, such as industrial machines and engines, but is now responsible for utilizing sensor data, developing software for anomaly detection and fault diagnosis, and verifying the practical effectiveness of distributed processing systems. Prior to IHI, he spent four years developing anomaly detection algorithms for machinery systems and social infrastructures at Kozo Keikaku Engineering Inc. Yoshitaka holds an MEng in aeronautics and astronautics from the University of Tokyo.