JD.com is China’s largest online retailer and its biggest overall retailer, as well as the country’s biggest Internet company by revenue. Currently, JD.com’s BDP platform running more than 400 thousand job (15PB+) daily, the number of all cluster’s node is more than 15000+, the total capacity is 210PB.
Alluxio, formerly Tachyon, is the world’s first system that unifies disparate storage systems at memory speed. In the big data ecosystem, Alluxio lies between computation frameworks or jobs and various kinds of storage systems. Additionally, Alluxio’s memory-centric architecture enables data access orders of magnitude faster than existing solutions.
Alluxio as a fault-tolerant pluggable optimization component is applicable to many computing framework of JD system. We can reduce dependence on network consumption by using Alluxio’s excellent caching capabilities to provide support for ADHOC and real-time stream computing. One of them, the JDPresto on Alluxio has led to a 10x performance improvement on average. Lastly, Alluxio is integrated as a pluggable optimization components and JDPresto can access to HDFS directly when Alluxio service is not available. Our work also extended Alluxio and enhanced the syncing between Alluxio and HDFS for consistency. Alluxio has run in our production envrionment on 100 nodes for 6 months.
Chinese. Focus on big data.
Zhehan Wang. Chinese. Focus big data.
Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?
Join the conversation here (requires login)
©2018, O’Reilly UK Ltd • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • email@example.com