Presented By O’Reilly and Cloudera
Make Data Work
21–22 May 2018: Training
22–24 May 2018: Tutorials & Conference
London, UK

Using Alluxio(formerly Tachyon) as a fault-tolerant pluggable optimization component to compute frameworks of JD system

12:0512:45 Wednesday, 23 May 2018

Who is this presentation for?

BI engineer, Distributed software developers

Prerequisite knowledge

Familiarity with Alluxio and HDFS

What you'll learn

Learn about how Alluxio as a pluggable optimization component and how we use Alluxio to provide support for ADHOC and real-time stream computing while keep the consistent of Alluxio and HDFS.

Description

JD.com is China’s largest online retailer and its biggest overall retailer, as well as the country’s biggest Internet company by revenue. Currently, JD.com’s BDP platform running more than 400 thousand job (15PB+) daily, the number of all cluster’s node is more than 15000+, the total capacity is 210PB.

Alluxio, formerly Tachyon, is the world’s first system that unifies disparate storage systems at memory speed. In the big data ecosystem, Alluxio lies between computation frameworks or jobs and various kinds of storage systems. Additionally, Alluxio’s memory-centric architecture enables data access orders of magnitude faster than existing solutions.

Alluxio as a fault-tolerant pluggable optimization component is applicable to many computing framework of JD system. We can reduce dependence on network consumption by using Alluxio’s excellent caching capabilities to provide support for ADHOC and real-time stream computing. One of them, the JDPresto on Alluxio has led to a 10x performance improvement on average. Lastly, Alluxio is integrated as a pluggable optimization components and JDPresto can access to HDFS directly when Alluxio service is not available. Our work also extended Alluxio and enhanced the syncing between Alluxio and HDFS for consistency. Alluxio has run in our production envrionment on 100 nodes for 6 months.

Photo of mao baolong

mao baolong

JD

Chinese. Focus on big data.

Photo of Wang Zhehan

Wang Zhehan

JD

Zhehan Wang. Chinese. Focus big data.

Leave a Comment or Question

Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?

Join the conversation here (requires login)