Doing a proof of concept with Elasticsearch and the Elastic stack is easy. Pushing the limits of its performance and scale is quite another thing. Radu Gheorghe and Rafał Kuć concentrate on the latter, discussing both the pitfalls and the best practices of using Elasticsearch for logs and metrics.
Radu and Rafał start by looking at how to scale Elasticsearch through a combination of time- and size-based indices and how to divide the cluster in tiers in order to handle the potentially spiky load in real time. They’ll focus largely on tuning individual nodes, covering everything from refreshes and flushes, buffers and caches, and merge policies and doc values to OS settings like disk scheduler, SSD caching, and huge pages. Some of these settings will be different for storing logs and metrics—Radu and Rafał explain how and why.
Radu and Rafał conclude with a look at the pipeline for getting the logs to Elasticsearch and demonstrate how to make it fast and reliable: where should buffers live, which protocols to use, where should the heavy processing be done (like parsing unstructured data), and which tools from the ecosystem can help.
Rafał Kuć is a search consultant and software engineer at Sematext Group, Inc. mainly focused on Lucene, Solr, Elasticsearch, Hadoop, and Mahout. Rafał is the author of the Apache Solr Cookbook series and Elasticsearch Server. He is a father, a consultant at Sematext, and cofounder of the blog solr.pl, where he tries to share his knowledge.
©2016, O’Reilly UK Ltd • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • firstname.lastname@example.org