Apache Spark plays a key role in addressing several big data challenges in Bing. The diverse set of capabilities in Spark enables a variety of internet-scale workloads that power Bing services. The value Spark adds to the business and how well it fits with the existing data platform architecture complementing existing internal and external big data frameworks is clearly the driver behind the adoption of Spark for various next-gen data processing investments in Bing.
Kaarthik Sivashanmugam shares the Bing team’s experiences with Spark, discussing how Spark is employed in the use cases and covering batch processing of document corpus spanning the web and near real-time processing of events corresponding to hundreds of millions of search queries. Kaarthik also explores the challenges the team faced in adopting Spark and implementing scalable data processing pipelines and explains how they influenced the team in customizing Spark and building extensions.
Kaarthik is a Principal Software Engineering Manager in the AI Platform group at Microsoft. In his current role, he is leading a team of software engineers and applied scientists in implementing large scale training workloads on Azure Machine Learning service and enhancing the service to make it the best cloud-platform for data scientists and ML engineers. Prior to this work, Kaarthik was involved in the development of near real time data processing platform and GPU infrastructure for deep learning.
©2017, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • email@example.com
Apache Hadoop, Hadoop, Apache Spark, Spark, and Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries, and are used with permission. The Apache Software Foundation has no affiliation with and does not endorse, or review the materials provided at this event, which is managed by O'Reilly Media and/or Cloudera.