Scaling Solr Horizontally in the Cloud

Data: Scaling
Location: B118-119
Average rating: ****.
(4.00, 1 rating)

Scaling Solr horizontally (having multiple indexes) can be painful, and may even cause nightmares in some situations. There are multiple techniques available to scale Solr horizontally. The simplest is running multiple Solr webapps, however this falls apart due to the memory requirements. Next up is using Solr’s multiple core functionality, which scales much further but still runs into memory usage issues after awhile. The final technique is to use the multiple core in combination with LRU (least recently used) management code to shutdown inactive cores. Lets dive into each of these techniques in more detail.

Running multiple Solr webapps is the first technique and easiest to setup. The problems is it doesn’t scale very far at all. It doesn’t take many instances of Solr webapps to be running before it starts to eat up a ton of memory. Adding swap only helps for so long, before your severs start to scream for mercy.

Multicore on the other hand, allows Solr to scale much further, especially if you can share a common schema between cores. Setting up cores and the stopping of cores is easily accomplished via HTTP requests.

If you have many inactive cores in your cloud you can scale Solr even further by managing the multiple core setup. If you have an application were the cores do not need to be active at all times you can implement a LRU management layer. When the index is needed it is started if it is stopped. The other half of it is a management script that finds the LRU cores and shuts them down.

Solr is a great platform to use for searching your application’s data, and knowing different techniques for scaling it horizontally will help you make the most of it within your cloud environment.

Photo of Andy Blyler

Andy Blyler

Barracuda Networks

Andy is a programmer, system administrator, entrepreneur, musician, private pilot, & optimist. He is the lead software engineer on the Barracuda Networks Backup product.

Photo of Lindsay Snider

Lindsay Snider

Comments on this page are now closed.


Picture of Sheeri K. Cabral
Sheeri K. Cabral
09/04/2011 10:00pm PDT

A video for this talk can be found at