Sep 23–26, 2019

Downscaling: The Achilles' heel of autoscaling Spark clusters

Prakhar Jain (Qubole), Sourabh Goyal (Qubole)
4:35pm5:15pm Wednesday, September 25, 2019
Location: 1A 21

Who is this presentation for?

Data Engineers




Adding nodes at runtime (Upscale) to already running Spark-on-Yarn clusters is fairly easy. But taking away these nodes (Downscale) when the workload is low at some later point of time in a difficult problem. To remove a node from a running cluster, We need to make sure that it is not used for compute as well as storage. But on production workloads, we see that many of the nodes can’t be taken away because:

  1. Nodes are running some containers although they are not fully utilized. That means all containers are fragmented on different nodes. Exa. – each node is running 1-2 containers/executors although they have resources to run 4 containers. Also long running Spark executors makes it even more difficult.
  2. Nodes have some shuffle data in the local disk which will be consumed by Spark application running on this cluster later. In this case, the Resource Manager will never decide to reclaim these nodes because losing shuffle data could lead to costly recomputation of stages or tasks.

In this talk we will talk about how we can improve downscaling in Spark-on-YARN clusters under the presence of such constraints. We will cover changes in scheduling strategy for container allocation in YARN and Spark task scheduler which together helps us achieve better packing of containers. This makes sure that containers are defragmented on fewer set of nodes and thus some nodes don’t have any compute. By being careful in how we assign containers in the first place, we can prevent the chance of running into situations where containers of the same application are running over different nodes. In addition to this, we will also cover enhancements to Spark driver and External Shuffle Service (ESS) which helps us to proactively delete shuffle data which we already know has been consumed. This makes sure that nodes are not holding any unnecessary shuffle data – thus freeing them from storage and hence available for reclamation for faster downscaling.

Prerequisite knowledge

Familiarity with cloud as a concept.

What you'll learn

Efficient downscaling techniques for elastic clusters on the cloud.
Photo of Prakhar Jain

Prakhar Jain


Prakhar Jain is a member of the technical staff at Qubole, where he works in Spark team. Prakhar holds a bachelor of computer science engineering from the Indian Institute of Technology, Bombay, India.

Sourabh Goyal


Sourabh Goyal is a member of the technical staff at Qubole, where he works in Hadoop team. Sourabh holds a bachelor in computer engineering from Netaji Shubas Institute of Technology, University of Delhi

Leave a Comment or Question

Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?

Join the conversation here (requires login)

Contact us

For conference registration information and customer service

For more information on community discounts and trade opportunities with O’Reilly conferences

For information on exhibiting or sponsoring a conference

Contact list

View a complete list of Strata Data Conference contacts