Get the free Ebook:
Private and Open Data in Asia: A Regional Guide.
We are developing a platform to process massive sensor data obtained from social infrastructures and industrial machinery all over the world, in order to achieve advanced safety management. To provide service for mobile objects, we are implementing into this platform algorithms and software to analyze large-scale GIS data.
We first focused on the analysis of time-series using over 100,000 mobile objects. Since the number of observations exceeds 1.5 billion records per year, the algorithm requires both scalability and flexibility. To address this issue, we evaluated Spark on YARN as a scalable and flexible platform.
In this session, we’ll talk about the capability of Spark to realize numerical data processing, the best practices for application development, and realistic lessons on operating Spark on YARN.
IHI is a leading manufacturer in Japan that provides a wide variety of products including energy plants and aircraft engines. IHI is addressing this project, and NTT Data is providing technical support for Spark on YARN infrastructure building.
Masaru Dobashi is a system infrastructure engineer and leads the OSS professional service team at NTT DATA Corporation. Masaru developed an enterprise Hadoop cluster consisting of over 1,000 nodes in 2009, which was one of the largest Hadoop clusters in Japan at the time. After that, he designed and provisioned several kinds of clusters using nonHadoop OSS, such as Spark and Storm. Masaru is now responsible for introducing Hadoop, Spark, Storm, and other OSS middlewares into enterprise systems and developing data processing systems.
Yoshitaka Suzuki is a researcher in information science and technology at IHI Corporation. Yoshitaka has developed anomaly detection algorithms for several kinds of products, such as industrial machines and engines, but is now responsible for utilizing sensor data, developing software for anomaly detection and fault diagnosis, and verifying the practical effectiveness of distributed processing systems. Prior to IHI, he spent four years developing anomaly detection algorithms for machinery systems and social infrastructures at Kozo Keikaku Engineering Inc. Yoshitaka holds an MEng in aeronautics and astronautics from the University of Tokyo.
©2015, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • email@example.com
Apache Hadoop, Hadoop, Apache Spark, Spark, and Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries, and are used with permission. The Apache Software Foundation has no affiliation with and does not endorse, or review the materials provided at this event, which is managed by O'Reilly Media and/or Cloudera.