Presented By O'Reilly and Cloudera
Make Data Work
Feb 17–20, 2015 • San Jose, CA

Going Beyond the Needle in a Haystack: Elasticsearch and the ELK Stack

Kurt Hurtado (Elasticsearch Inc), Tal Levy (Elasticsearch)
1:30pm–5:00pm Wednesday, 02/18/2015
Data Science
Location: 210 B/F
Average rating: ****.
(4.50, 4 ratings)

Materials or downloads needed in advance

* Laptop, preferably Mac or Linux running on a virtual machine * Developer credentials for Twitter account preferred



The canonical challenge of finding a needle in a haystack is comparatively easy to finding out if there are other needles in the haystack, what their average length is, and where they tend to group together. With the combined power of Elasticsearch, Logstash, and Kibana, the three massively popular open source projects that make up the ELK stack, this kind of data exploration is possible.

This tutorial will provide an introduction to the individual components of the ELK stack followed by a discussion of use cases and a hands-on laboratory. This includes installing and configuring Elasticsearch, Logstash, and Kibana. The instructors will cover guidelines and tips on processing custom log formats, designing a system to scale, choosing hardware, creating real-time dashboards, and managing the lifecycle of your logs.

Throughout this session, we will discuss the exploration of a broad range of data — from Apache logs to Twitter streams — and collectively solve a data analytics challenge.


  • Please bring a laptop, preferably Mac or Linux running on a virtual machine.
  • Developer credentials for Twitter account preferred.
  • Please download this file before attending. We’ll be using the software and data files contained in this download.

Kurt Hurtado

Elasticsearch Inc

Kurt Hurtado is a Logstash developer based in Los Altos, CA. He has been working with Elasticsearch and Logstash for many years and thrives on building excellent architectures based on the ELK stack for customers and internal to Elasticsearch. Prior to Elasticsearch, Kurt has performed various development and operations roles in startups as well as large enterprises.

Tal Levy


Comments on this page are now closed.


Kurt Hurtado
02/18/2015 8:28am PST

Thanks for attending! The slides are available here:

Gal Heyne
02/18/2015 6:17am PST

Are the slides available to download?

Kurt Hurtado
02/17/2015 1:13am PST

We are looking forward to seeing everyone at STRATA!

It would be helpful if you could download the following file before attending:

We’ll be using the software and data files contained in this download.

Kurt Hurtado
02/12/2015 4:26am PST

Harish: 4GB+ should be fine. 2GB might work ok too.

Deepali: we’d highly recommend running a linux VM on your windows machine. Any chance of that? Windows support for Logstash will be much better in the upcoming 1.5 release!

If you’d like to work on the twitter demo, you should visit twitter’s dev site beforehand, create an application, and generate the keys and access tokens:

You’ll need the following from twitter:

  • Consumer Key (API Key)
  • Consumer Secret (API Secret)
  • Access Token
  • Access Token Secret

See you there!

Harish Thilliyambur Krishnan
02/11/2015 3:07am PST

Is there a minimum RAM requirement for Mac?

Deepali Bhandari
01/16/2015 3:57am PST

Will windows 7 64 bit 4gb ram be sufficient for this tutorial. Please advise