Sep 23–26, 2019

Carson Wang
Big Data Software Engineer, Intel

Carson Wang is a big data software engineer at Intel, where he focuses on developing and improving new big data technologies. He’s an active open source contributor to the Apache Spark and Alluxio projects as well as a core developer and maintainer of HiBench, an open source big data microbenchmark suite. Previously, Carson worked for Microsoft on Windows Azure.

Sessions

5:25pm–6:05pm Wednesday, September 25, 2019

Location: 1A 21/22

Improving Spark by taking advantage of disaggregated architecture

Data Engineering and Architecture

Secondary topics: Data, Analytics, and AI Architecture, Deep dive into specific tools, platforms, or frameworks

Chenzhao Guo (Intel), Carson Wang (Intel)

Shuffle in Spark requires the shuffle data to be persisted on local disks. However, the assumptions of collocated storage do not always hold in today’s data centers. Chenzhao Guo and Carson Wang outline the implementation of a new Spark shuffle manager, which writes shuffle data to a remote cluster with different storage backends, making life easier for customers. Read more.