Elasticsearch is a highly scalable NoSQL document store specifically leveraging Lucene indexes in order to allow for deep data introspection. Elasticsearch is already the de facto system to use for log analysis but has recently branched out into time series data manipulation and analysis. Christian Saide explains how NS1 was able to reduce infrastructure, maintenance, and operational costs while simultaneously increasing throughput and visibility of key metrics by leveraging Elasticsearch as a time series database.
NS1 historically used a time series database to do its operational metrics analysis, alongside Elasticsearch to do log analysis. This time series database and its supporting architecture quickly grew to the point where NS1 needed dedicated team members to manage it. This, coupled with the fact that NS1 also had an Elasticsearch cluster to manage, forced the company to rethink its solution. It needed to ensure the metrics throughput the current time series database would be supported, which at the time was in the rage of 150–200 thousand points per second ingested. Using a small set of 10 servers running its Elasticsearch cluster, NS1 was able to achieve throughput numbers of 650–700 thousand documents per second indexed, which proved that NS1 could and more importantly should combine the two systems.
The deep data introspection offered by Elasticsearch is the key differentiator when compared to other classical time series databases. Due to its introspection capabilities, an operator is given the tools to allow for making connections that a standard time series database would not traditionally allow for. These capabilities are amplified by dramatically reducing operational burden through a thriving community of plugins and support networks. The combination of data introspection and lighter operational overhead enables operations teams to have more throughput and allows for easier access to the key data that they need to operate distributed infrastructure. This solution has the added benefit of also reducing the infrastructure and maintenance costs of operating two standalone pieces of technology.
Christian Saide is a DevOps engineer at NS1, where he has been a key player in automating, hardening, and scaling out its systems, particularly by pushing more and more of its infrastructure into container-based architectures and implementing solutions to the tough problems surrounding global distribution. He also served a critical role in NS1’s move to software-defined networking and authored the primary software-defined networking device and network topology. Christian has been working in the technology sector for five years, focusing on networking and distributed systems. Previously, he was at Industrial Color Software, where he climbed from a midlevel software developer to director of development operations and was instrumental in taking the company’s aging infrastructure from a handful of bare-metal servers to multiple virtualization hosts running hundreds of virtual machines, which in turn supported hundreds of containers.
©2018, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • email@example.com