Build Systems that Drive Business
June 11–12, 2018: Training
June 12–14, 2018: Tutorials & Conference
San Jose, CA

Gaining efficiency with time series in ELK

1:15pm–1:55pm Wednesday, June 13, 2018
Monitoring, Observability, and Performance
Location: LL21 A/B Level: Beginner
Secondary topics: Systems Monitoring & Orchestration
Average rating: ****.
(4.33, 3 ratings)

Prerequisite knowledge

  • A general understanding of time series databases and their architecture
  • Familiarity with Elasticsearch

What you'll learn

  • Learn how NS1 was able to reduce infrastructure, maintenance, and operational costs while simultaneously increasing throughput and visibility of key metrics by leveraging Elasticsearch as a time series database

Description

Elasticsearch is a highly scalable NoSQL document store specifically leveraging Lucene indexes in order to allow for deep data introspection. Elasticsearch is already the de facto system to use for log analysis but has recently branched out into time series data manipulation and analysis. Christian Saide explains how NS1 was able to reduce infrastructure, maintenance, and operational costs while simultaneously increasing throughput and visibility of key metrics by leveraging Elasticsearch as a time series database.

NS1 historically used a time series database to do its operational metrics analysis, alongside Elasticsearch to do log analysis. This time series database and its supporting architecture quickly grew to the point where NS1 needed dedicated team members to manage it. This, coupled with the fact that NS1 also had an Elasticsearch cluster to manage, forced the company to rethink its solution. It needed to ensure the metrics throughput the current time series database would be supported, which at the time was in the rage of 150–200 thousand points per second ingested. Using a small set of 10 servers running its Elasticsearch cluster, NS1 was able to achieve throughput numbers of 650–700 thousand documents per second indexed, which proved that NS1 could and more importantly should combine the two systems.

The deep data introspection offered by Elasticsearch is the key differentiator when compared to other classical time series databases. Due to its introspection capabilities, an operator is given the tools to allow for making connections that a standard time series database would not traditionally allow for. These capabilities are amplified by dramatically reducing operational burden through a thriving community of plugins and support networks. The combination of data introspection and lighter operational overhead enables operations teams to have more throughput and allows for easier access to the key data that they need to operate distributed infrastructure. This solution has the added benefit of also reducing the infrastructure and maintenance costs of operating two standalone pieces of technology.

Topics include:

  • The benefits of utilizing Elasticsearch to analyze time series data sources side by side with unstructured data like logs
  • Configuration and throughput of NS1’s current cluster, in terms of how much data it is able to collect, analyze, and store
  • Key disadvantages of Elasticsearch for metrics analysis
Photo of Christian Saide

Christian Saide

NS1

Christian Saide is a DevOps engineer at NS1, where he has been a key player in automating, hardening, and scaling out its systems, particularly by pushing more and more of its infrastructure into container-based architectures and implementing solutions to the tough problems surrounding global distribution. He also served a critical role in NS1’s move to software-defined networking and authored the primary software-defined networking device and network topology. Christian has been working in the technology sector for five years, focusing on networking and distributed systems. He spent the first half of his career at Industrial Color Software, where he climbed from a midlevel software developer to director of development operations. Throughout his tenure, he was instrumental in taking the company’s aging infrastructure from a handful of bare-metal servers to multiple virtualization hosts running hundreds of virtual machines, which in turn supported hundreds of containers.