Presented By O'Reilly and Cloudera
Make Data Work
September 25–26, 2017: Training
September 26–28, 2017: Tutorials & Conference
New York, NY

MacroBase: A search engine for fast data streams

Sahaana Suri (Stanford University)
2:55pm3:35pm Thursday, September 28, 2017
Stream processing and analytics
Location: 1E 15/16 Level: Intermediate
Secondary topics:  Streaming
Average rating: *****
(5.00, 1 rating)

Who is this presentation for?

  • Data engineers, DevOps engineers, and data scientists

Prerequisite knowledge

  • A basic understanding of data analytics pipelines
  • Production experience with time series or data science (useful but not required)

What you'll learn

  • Explore MacroBase, a new analytics engine from Stanford designed to prioritize the scarcest resource in large-scale, fast-moving data streams: human attention


MacroBase is a new open source analytics engine from the Stanford InfoLab designed to prioritize the scarcest resource in large-scale, fast-moving data streams: human attention. In many deployments at scale, an overwhelming proportion of data collected is never read and is instead retained only for reactive failure analysis. MacroBase analyzes data as it arrives, providing high-level interpretable explanations of stream behaviors, thus increasing its utility and enabling real-time root-cause analysis and anomaly detection.

At its core, MacroBase combines streaming classification and explanation operators to both identify individual points of interest and highlight commonalities across them. For example, the Android device ecosystem comprises over 24,000 distinct device types. How can you determine whether your mobile application is behaving correctly on all of them? MacroBase’s classification operators can identify abnormally behaving devices, while its explanation operators can aggregate many such devices, producing more interpretable outputs. Thus, MacroBase is designed as both a set of reconfigurable dataflow operators as well as a series of end-to-end dataflow pipelines that have already been used to diagnose issues in production streams in mobile, data center, and industrial applications.

Sahaana Suri walks you through the core concepts behind MacroBase, its architecture, and key use cases and shares takeaways from the recent research literature for data scientists, data engineers, and DevOps engineers.

Photo of Sahaana Suri

Sahaana Suri

Stanford University

Sahaana Suri is a second year PhD student in the Stanford InfoLab, working with Peter Bailis. Sahaana’s research focuses on building easily accessible data analytics and machine learning systems that scale. She holds a bachelor’s degree in electrical engineering and computer science from the University of California, Berkeley.