MacroBase is a new open source analytics engine from the Stanford InfoLab designed to prioritize the scarcest resource in large-scale, fast-moving data streams: human attention. In many deployments at scale, an overwhelming proportion of data collected is never read and is instead retained only for reactive failure analysis. In response, MacroBase analyzes data as it arrives, providing high-level interpretable explanations of stream behaviors, thus increasing its utility and enabling real-time, root cause analysis and anomaly detection.
At its core, MacroBase combines streaming classification and explanation operators to both identify individual points of interest as well as highlight commonalities across them. For example, the Android device ecosystem comprises over 24,000 distinct device types; is a mobile application behaving correctly on all of them? MacroBase’s classification operators can identify abnormally behaving devices, while its explanation operators can aggregate many such devices, producing more interpretable outputs. Thus, MacroBase is designed as both a set of reconfigurable dataflow operators as well as a series of end-to-end dataflow pipelines that have already been used to diagnose issues in production streams in mobile, data center, and industrial applications.
We will walk through the core concepts behind MacroBase, its architecture, key use cases, and takeaways from the recent research literature for data scientists, data engineers, and DevOps.
Sahaana Suri is a second year PhD student in the Stanford InfoLab, working with Peter Bailis. Sahaana’s research focuses on building easy-to-use, accessible data analytics and machine learning systems that scale. She holds a bachelor’s degree in Electrical Engineering and Computer Science from the University of California, Berkeley.
Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?
Join the conversation here (requires login)
©2017, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • email@example.com