October 30–31, 2016: Training
October 31–November 2, 2016: Tutorials & Conference
New York, NY

Building effective security alerting

Kenneth Lee (Etsy), Kai Zhong (Etsy)
11:20am–12:00pm Wednesday, 11/02/2016
Tools and processes
Location: Grand Ballroom West Level: Intermediate
Average rating: ****.
(4.00, 10 ratings)

Prerequisite knowledge

  • A general familiarity with ELK and experience with, or at least an interest in, setting it up as a log analysis engine

What you'll learn

  • Understand how to leverage 411 to use the search capabilities of Elasticsearch to build effective alerting
  • Learn how to use and extend 411 to bring its alerting capabilities to other systems in your infrastructure


Modern web applications generate a ton of logs. Suites like ELK (Elasticsearch, Logstash, and Kibana) exist to help manage these logs, and more people are turning to them for their log analysis needs. These logs contain a treasure trove of information regarding bad actors on your site, but surfacing that information in a timely manner can be difficult. When Etsy moved over from Splunk to ELK in mid-2014, it realized that ELK lacked necessary functionality to allow real-time alerting. Etsy was in need of a solution that provided a robust means of querying ELK and adding additional context to the data. It ended up creating its own framework to provide this functionality.

Kenneth Lee and Kai Zhong introduce 411, Etsy’s new open source tool designed as a solution for detecting and alerting on interesting anomalies and security events. The Security team at Etsy was interested in using this functionality to detect everything from XSS to monitoring for potential account compromises. Kenneth and Kai start with a discussion of what you should be logging into Elasticsearch. This is important to help you create useful, actionable alerts in 411. They note a number of configuration tips and tricks to help you get the most out of your ELK cluster before diving into 411’s features and explaining how it allows the Etsy Security team to work effectively. Kenneth and Kai conclude with two demos of 411 in action. Along the way, they demonstrate several examples of useful searches you can build in 411 and show how this data can be manipulated to generate clear actionable alerts; they also explore the built-in workflow for responding to alerts and explain how 411 allows you to pull up additional context as you work on an alert.

While much of the discussion will be centered around ELK, 411 can be used with a variety of data sources in parallel. 411 is open source, and includes several of these search plugins. You’ll learn how you can make use of these plugins to immediately start building alerts and how to build additional extensions to connect 411 to new data sources.

Photo of Kenneth Lee

Kenneth Lee


Kenneth Lee is a senior product security engineer at Etsy working on everything from managing the bug bounty program to shattering the site with new vulnerabilities. Previously, Kenneth worked at FactSet Research Systems preventing hackers from stealing financial data. He holds an MS in computer science with a focus on computer security from Columbia. Between sweet hacks, Kenneth enjoys drinking tea and force-feeding Etsy’s operations team Japanese chocolates.

Photo of Kai Zhong

Kai Zhong


Kai Zhong is a security engineer at Etsy. At work, he maintains security features on the site, works on 411, does the occasional code review, and breaks the login page. Kai spends his free time playing video games, participating in CTFs, and trying to learn abstract algebra. He prefers cats, but dogs are fine too.