Presented By O’Reilly and Cloudera
Make Data Work
21–22 May 2018: Training
22–24 May 2018: Tutorials & Conference
London, UK

Batch and real-time processing in LINE's log analysis platform

Wataru Yukawa (LINE)
17:25–18:05 Wednesday, 23 May 2018
Data engineering and architecture
 Location: Capital Suite 2/3 Level: Beginner
View slides

Who is this presentation for?

  • Data engineers

Prerequisite knowledge

  • Familiarity with batch and real-time processing concepts and techniques

What you'll learn

  • Explore LINE's web tracking system

Description

LINE—one of the most popular messaging applications in Asia—offers many services, such as its news application. These services sometimes depend on real-time processing.

Wataru Yukawa offers an overview of LINE’s two-layer log analysis platform, consisting of a batch layer, which uses Hive, Presto, and Hadoop, and a web tracking system, which uses the JavaScript SDK, NGINX Fluentd, Kafka, Elasticsearch, and Hadoop. Wataru focuses on the latter to explain how it helps with batch and real-time processing.

Photo of Wataru Yukawa

Wataru Yukawa

LINE

Wataru Yukawa is a data engineer at LINE, where he is creating and maintaining a log analysis platform based on Hadoop, Hive, Fluentd, Presto, and Azkaban and working on aggregating log and RDBMS data with Hive and reporting using BI tools.

Comments on this page are now closed.

Comments

Picture of Wataru Yukawa
Wataru Yukawa | DATA ENGINEER
22/05/2018 19:42 BST

My slide is https://www.slideshare.net/wyukawa/strata2018london-98116246