MacroBase is a new open source analytics engine from the Stanford InfoLab designed to prioritize the scarcest resource in large-scale, fast-moving data streams: human attention. In many deployments at scale, an overwhelming proportion of data collected is never read and is instead retained only for reactive failure analysis. MacroBase analyzes data as it arrives, providing high-level interpretable explanations of stream behaviors, thus increasing its utility and enabling real-time root-cause analysis and anomaly detection.
At its core, MacroBase combines streaming classification and explanation operators to both identify individual points of interest and highlight commonalities across them. For example, the Android device ecosystem comprises over 24,000 distinct device types. How can you determine whether your mobile application is behaving correctly on all of them? MacroBase’s classification operators can identify abnormally behaving devices, while its explanation operators can aggregate many such devices, producing more interpretable outputs. Thus, MacroBase is designed as both a set of reconfigurable dataflow operators as well as a series of end-to-end dataflow pipelines that have already been used to diagnose issues in production streams in mobile, data center, and industrial applications.
Sahaana Suri walks you through the core concepts behind MacroBase, its architecture, and key use cases and shares takeaways from the recent research literature for data scientists, data engineers, and DevOps engineers.
Sahaana Suri is a second year PhD student in the Stanford InfoLab, working with Peter Bailis. Sahaana’s research focuses on building easily accessible data analytics and machine learning systems that scale. She holds a bachelor’s degree in electrical engineering and computer science from the University of California, Berkeley.
©2017, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • firstname.lastname@example.org