Downscaling: The Achilles heel of autoscaling Spark Clusters
Who is this presentation for?Data Engineers
Prerequisite knowledgeFamiliarity with cloud as a concept.
What you'll learn
Adding nodes at runtime (Upscale) to already running Spark-on-Yarn clusters is fairly easy. But taking away these nodes (Downscale) when the workload is low at some later point of time in a difficult problem. To remove a node from a running cluster, We need to make sure that it is not used for compute as well as storage. But on production workloads, we see that many of the nodes can’t be taken away because:
- Nodes are running some containers although they are not fully utilized. That means all containers are fragmented on different nodes. Exa. – each node is running 1-2 containers/executors although they have resources to run 4 containers. Also long running Spark executors makes it even more difficult.
- Nodes have some shuffle data in the local disk which will be consumed by Spark application running on this cluster later. In this case, the Resource Manager will never decide to reclaim these nodes because losing shuffle data could lead to costly recomputation of stages or tasks.
In this talk we will talk about how we can improve downscaling in Spark-on-YARN clusters under the presence of such constraints. We will cover changes in scheduling strategy for container allocation in YARN and Spark task scheduler which together helps us achieve better packing of containers. This makes sure that containers are defragmented on fewer set of nodes and thus some nodes don’t have any compute. By being careful in how we assign containers in the first place, we can prevent the chance of running into situations where containers of the same application are running over different nodes. In addition to this, we will also cover enhancements to Spark driver and External Shuffle Service (ESS) which helps us to proactively delete shuffle data which we already know has been consumed. This makes sure that nodes are not holding any unnecessary shuffle data – thus freeing them from storage and hence available for reclamation for faster downscaling.
Prakhar Jain is a member of the technical staff at Qubole, where he works in Spark team. Prakhar holds a bachelor of computer science engineering from the Indian Institute of Technology, Bombay, India.
Sourabh Goyal is a member of the technical staff at Qubole, where he works in Hadoop team. Sourabh holds a bachelor in computer engineering from Netaji Shubas Institute of Technology, University of Delhi
Leave a Comment or Question
Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?
Join the conversation here (requires login)
For conference registration information and customer service
For more information on community discounts and trade opportunities with O’Reilly conferences
For information on exhibiting or sponsoring a conference
View a complete list of Strata Data Conference contacts