Spark SQL—the most popular component of Apache Spark—is widely used to process large-scale structured data in the data center. However, it still suffers from stability and performance challenges in highly dynamic environments with ultra-large-scale data.
Haifeng Chen shares a Spark adaptive execution engine built to address these challenges. It can handle task parallelism, join conversion, and data skew dynamically during runtime, guaranteeing the best plan is chosen using runtime statistics, and has provided significant performance improvements in typical SQL benchmarks like TPC-DS. The performance of this approach has been proven by its adoption by a number of Chinese internet companies.
Haifeng details the major three challenges the industry faces when using Spark SQL in real-world environments, outlines the technical architecture of the adaptive execution approach as well as the technical details of each solution designed to solve these challenges, and shares benchmark results and experiences from industrial adoptions. Haifeng concludes by discussing planned advances to optimize the Spark Adaptive Execution engine and take it to the next level.
Haifeng Chen is a senior software architect at Intel’s Asia Pacific R&D Center. He has more than 12 years’ experience in software design and development, big data, and security, with a particular interest in image processing. Haifeng is the author of image browsing, editing, and processing software ColorStorm.
©2019, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • firstname.lastname@example.org