Alluxio—the first memory-speed virtual distributed storage system in the world—unifies the data from various under storage systems and presents a global namespace to various computation frameworks. Data access can be several magnitudes faster because of Alluxio’s memory-centric architecture. In addition, Alluxio’s tiered storage, unified namespace, flexible file API, web UI, and command-line tools increase the usability in different application scenarios. The Alluxio open source project is one of the fastest growing big data projects, with more than 600 contributors from more than 100 companies across the world.
Qunar is the number-one Chinese-language online travel information provider and search engine for web-based and mobile users. Currently, Qunar’s streaming platform processes around 6 billion system log entries (~4.5 TB) daily. Many jobs running on the platform are business critical and therefore impose strict requirements on both stability and low latency. For example, real-time user recommendations are generated mainly based on the log analysis of a user’s click behavior as well as the search pattern. The faster the iteration of the analysis, the more accurate the feedback that Qunar can deliver to the users. Therefore low latency and high stability are the top priorities of its system.
Xueyan Li and Yupeng Fu explore how Alluxio has led to performance improvements averaging a 300x improvement at service peak time on stream processing workloads at Qunar.
Xueyan Li is a data platform R&D engineer at Qunar, where he is mainly responsible for the continuous integrated development of resource management system Mesos and distributed memory management system Alluxio, as well as data for all business lines based on public service support. Other focuses include the ELK log ETL platform, Spark, Storm, Flink, and Zeppelin. He holds a degree in software engineering from Heilongjiang University.
Yupeng Fu is a founding member and senior architect at Alluxio and a PMC member of the Alluxio open source project. Previously, Yupeng worked at Google building big data analytics platforms and Palantir, where he led the efforts building the company’s storage solution. Yupeng holds a BS and an MS from Tsinghua University and has completed coursework toward a PhD at UCSD.
©2017, O’Reilly UK Ltd • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • firstname.lastname@example.org