Build resilient systems at scale
October 12–14, 2015 • New York, NY

From zero to production hero: Log analysis with Elasticsearch

Radu Gheorghe (Sematext Group), Rafał Kuć (Sematext Group)
1:30pm–3:00pm Monday, 10/12/2015
Tutorial
Location: Nassau Suite
Average rating: ***..
(3.55, 11 ratings)
Slides:   external link

Prerequisite Knowledge

Familiarity with syslog format and protocols, logstash and Elasticsearch would be nice, but we'll cover the essentials here, too.

Materials or downloads needed in advance

We'll share all configuration snippets on GitHub, so you can replay any parts that apply to your use-case.

Description

Elasticsearch has become the go-to data store for logs, because it allows you to search and analyze tons of data in milliseconds. Using tools like Logstash and Kibana, you can start feeding logs and graphing them in a few minutes.

As with all software dealing with lots of data, the road from PoC to production is often bumpy, as you might need to scale out to many nodes, and try to squeeze the last ounce of performance out of them.

This talk will take you from the basics of centralizing logs in Elasticsearch, to all the strategies that make it scale with billions of documents in production. We’ll cover:

  • Time-based indices and index templates to efficiently slice your data
  • Different node tiers to de-couple reading from writing, heavy traffic from low traffic
  • Tuning various Elasticsearch and OS settings to maximize throughput and search performance
  • Configuring tools such as logstash and rsyslog to maximize throughput and minimize overhead
Photo of Radu Gheorghe

Radu Gheorghe

Sematext Group

Radu Gheorghe is a search consultant and software engineer at Sematext, working mainly with Elasticsearch- and logging-related projects. He is the coauthor of Elasticsearch in Action.

Photo of Rafał Kuć

Rafał Kuć

Sematext Group

Rafał Kuć is a search consultant and software engineer at Sematext Group, Inc. mainly focused on Lucene, Solr, Elasticsearch, Hadoop, and Mahout. Rafał is the author of the Apache Solr Cookbook series and Elasticsearch Server. He is a father, a consultant at Sematext, and cofounder of the blog solr.pl, where he tries to share his knowledge.

Comments on this page are now closed.

Comments

Picture of Radu Gheorghe
Radu Gheorghe
10/28/2015 11:52pm EDT

Hi Michael,

Sure, you can find the slides and, more importantly, all the configs and code samples on Sematext’s blog

Any questions or feedback are welcome, of course :)

Michael Rivera
10/28/2015 2:07pm EDT

Can you provide the location of where your powerpoint was? I would like to review it. Thanks

Stay Connected

Follow Velocity on Twitter Facebook Group Google+ LinkedIn Group

Videos

More Videos »

O’Reilly Media

Tech insight, analysis, and research