Skip to main content

Machine Learning for Machine Data

David Andrzejewski (Sumo Logic)
Machine Data
Mission City M
Average rating: ****.
(4.67, 6 ratings)
Slides:   1-PDF 

Computational devices and systems are constantly telling their stories in the form of logs: logins and logouts, network connections and disconnections, sensor readings, customer interactions, database activities, configuration changes, and application errors. These logs can often tell us important information about the operational health, security, and usage patterns of our systems.

However, the scale and variety of raw log data can pose a formidable challenge. Logs come in many different formats, and these formats themselves evolve over time. For large log volumes, interactive single-machine processing is not an option. Random fluctuations and transient disruptions can make static rules and thresholds prone to “false positive” alerts.

In this talk we will discuss a few ways in which machine learning techniques can be combined with human guidance in order to understand what the logs are telling us:

  • detection of system-wide changes in behavior
  • “learning by example” to identify events
  • partially supervised discovery of log structure
  • inferring log relevance
  • graph mining of logs
  • time-series modeling of log metrics

We’ll also give examples of interesting findings uncovered by these applying these approaches to large volumes of real data in a production log-management service.

Photo of David Andrzejewski

David Andrzejewski

Lead Data Sciences Engineer, Sumo Logic

David Andrzejewski is Lead Data Sciences Engineer at Sumo Logic, which he joined in 2011 after a postdoctoral research position working on knowledge discovery at Lawrence Livermore National Laboratory (LLNL). He is interested in developing tools that combine the power of machine learning with human insights, and has published work applying these ideas to problems in biomedical text mining, information retrieval and software behavior. David completed his PhD in Computer Sciences at the University of Wisconsin-Madison in 2010, where he had also previously received an M.S. in Computer Sciences and a B.S. in Computer Engineering, Mathematics and Computer Sciences.

Comments on this page are now closed.

Comments

Picture of David Andrzejewski
David Andrzejewski
04/04/2014 2:54am PDT

Video of the talk posted: https://www.youtube.com/watch?v=WrRDK5I9iyw