Presented By O’Reilly and Cloudera
Make Data Work
March 5–6, 2018: Training
March 6–8, 2018: Tutorials & Conference
San Jose, CA

Spark on Kubernetes: A case study from JD.com

Zhen Fan (JD.com), Wei Ting Chen (Intel Corporate)
1:50pm2:30pm Wednesday, March 7, 2018
Average rating: ****.
(4.00, 4 ratings)

Who is this presentation for?

  • Big data and DevOps engineers

Prerequisite knowledge

  • A basic understanding of Spark, Docker and container concepts, and Kubernetes

What you'll learn

  • Learn how to run Spark on Kubernetes for big data and AI workloads
  • Explore a case study from JD.com

Description

Docker helps to easily build, ship, and run your applications. There are several ways to run Spark with Docker. Spark on Kubernetes is one of the solutions to run Spark by using a native way like Spark on Yarn or Mesos. Zhen Fan and Wei Ting Chen explain how JD.com uses Spark on Kubernetes in a production environment and why the company chose Spark on Kubernetes for its AI workloads. You’ll learn how to run with Spark with Docker while using Kubernetes as a resource manager and gain an understanding of the advantages this provides.

Topics include:

  • The differences between several Spark-on-Docker solutions
  • How Spark on Kubernetes works
  • The advantages for Spark on Kubernetes
  • A case study from JD.com on how to implement and use Spark on Kubernetes in a production environment
  • Performance results for Spark on Kubernetes
Photo of Zhen Fan

Zhen Fan

JD.com

Zhen Fan is a software development engineer at JD.com, where he focuses on machine learning platform development and management.

Photo of Wei Ting Chen

Wei Ting Chen

Intel Corporate

Wei Ting Chen is a senior software engineer in Intel’s Software Service Group, where he works on big data on cloud solutions. One of his responsibilities is helping customers integrate big data solutions into their cloud infrastructure. Wei Ting is a contributor to the OpenStack Sahara project.

Comments on this page are now closed.

Comments

Picture of Wei Ting Chen
Wei Ting Chen | SENIOR SOFTWARE ENGINEER
03/12/2018 7:51am PDT

I have uploaded the slides, please feel free to download the slides and ask any question.

Picture of Zhen Fan
Zhen Fan | SOFTWARE DEVELOPMENT ENGINEER
03/08/2018 6:31am PST

@Dean Sure, could you leave your email address, I’ll post you soon.

Picture of Dean Wampler
Dean Wampler | VICE PRESIDENT, FAST DATA ENGINEERING
03/07/2018 11:04pm PST

I enjoyed your presentation yesterday. Could you post the slides?